Research on Intelligent Synthesis 
Environment 

NASA Cooperative Agreement NCC-1-01040 
(ODURF Project 113282) 

Final Report of Subproject 

Dr. R. Bowen Loftin, co-Principal 

Investigator 

Dr. David Dryer, co-investigator 
Dr. Debra Major, co-investigator 
Mr. Thomas Fletcher, Graduate Research 

Assistant 


October 28, 2002 

Virginia Modeling, Analysis & Simulation Center 
Old Dominion University 


Research on Intelligent Synthesis Environment 
NASA Cooperative Agreement NCC-1-01040 
(ODURF Project 113282) 

Final Report of Subproject 

Dr. R. Bowen Loftin, co-Principal Investigator 
Dr. David Dryer, co-investigator 
Dr. Debra Major, co-investigator 
Mr. Thomas Fletcher, Graduate Research Assistant 

October 28, 2002 


Introduction 


The subproject described in the following report was funded at $80,00 0 and addressed 
^spssment and continuous improve men t of engineering team effectiveness in distnbu M 
collaborative enviro nments . The work described below was earned out from Apnl 1, 2001 
through September 30, 2002 by the research team listed in the next section. 


Personnel 


Dr. R. Bowen Loftin, co-Principal Investigator 

Dr. David Dryer, co-investigator 

Dr. Debra Major, co-investigator 

Mr. Tom Fletcher, Graduate Research Assistant 


Research Approach 

The research team initially met with NASA Langley Research Center (NAS^LaRQpereomiel 
to achieve a consensus on the proposed project plan and to obtatn guidance from NASA/LaRC 
regarding a specific NASA collaborative engineering project that would be a suitable can ^' 
for data collection. NASA LaRC recommended the Inter-center Systems Analysis Team (1SAT) 
as a focus for our data collection efforts. 

It was generally agreed that the research team would begin by analyzing the relevant literature on 
collaboration with a focus on team performance in collaborative engineering. The 
be followed by observations of ISAT activity within the context of a specific project. Following 
this data collection effort, an engineering team process model would bedevelopedforthe 
environment. Finally, this model would be used to develop recommendations for NASA/LaRC 
for the creation of virtual collaborative environments that could be applied to the case of e 
ISAT or similar efforts at distributed collaborative engineering teams conducting design and/or 
analysis of complex engineering systems or a system of systems. 



Research Results 


A detailed review of the literature of team performance was conducted and an assessment of this 
literature from the standpoint of distributed teams engaged in engineering design and analysis 
was performed. The results of this effort are documented in Annex A (A Review of Key Team 
Performance Processes: Implications for Engineering in Distributed Collaborative 
Environments). 

During August, 2001 the research team observed the IS AT within the context of a specific 
project. The results of these observations are contained in Annex B (Team Task Analysis of 
ISAT - Inter-center Systems Analysis Team). 

Based on both the literature review and the ISAT observations, an engineering team project 
model for a distributed collaborative environments was developed. This product is included in 
Annex C (Development and Assessment of an Engineering Team Process Model in Distributed 
Collaborative Environment: The Case of ISAT). 

Following the development of the model described in Annex C, the model and the results of the 
ISAT observations recorded in Annex B were used to prepare a final recommendation for the use 
of distributed collaborative environments for engineering design and analysis. The 
recommendation was further refined to specifically apply to the ISAT environment in use within 
NASA. Annex D (Virtual Collaborative Environments for System of Systems Engineering and 
Applications for ISAT) documents this recommendation. 

In addition to the four products contained in the annexes to this report, at least on journal 
publication is anticipated as a result of this research. In addition, a portion of the work reported 
here serves as the basis for the master’s thesis research of a student of Old Dominion University. 



Annex A 


A Review of Key Team Performance Processes: 

Implications for Engineering in Distributed Collaborative Environments 
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Executive Summary 

The ultimate goal of this research project is to develop a methodology for the assessment and 
continuous improvement of engineering team effectiveness in distributed collaborative 
environments. This review provides the theoretical foundation upon which subsequent empirical 
work will be based. Our review of the team performance literature has identified the following 
12 conceptually distinct team interaction processes as characteristic of effective teams (see Table 
1 on pp. 39-40 for definitions and descriptions). 


• 

Mission Analysis 

• 

Team Orientation 

• 

Resource Distribution 

• 

Communication 

• 

Leadership 

• 

Coordination 

• 

Timing 

• 

Mutual Performance Monitoring 

• 

Intra-team Feedback 

• 

Back-up Behaviors 

• 

Motivational Functions 

• 

Cooperation 


In addition, this review summarizes how team task characteristics (i.e., task type, task 
complexity, motivation, and temporal changes), team characteristics (i.e., team structure and 
team knowledge), and individual team member characteristics (i.e., dispositions and teamwork 
knowledge, skills, and abilities) affect team interaction processes, determine the relevance of 
these processes, and influence team performance. The costs and benefits of distributed team 
collaboration are also considered. The review concludes with a brief discussion of the nature of 


collaborative team engineering tasks. 
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Introduction 

The world is growing seemingly smaller as technology advances and organizations span 
greater geographic distances. Teams are often used in the workplace due to task demands (i.e., an 
individual could not complete the task alone) and a need to remain competitive in tight markets 
(Ilgen, Major, Hollenbeck & Sego, 1993). Increasingly, teams are geographically disbursed and 
must conduct their functions across time and space (Maznevski & Chudoba, 2000). Travel is 
costly and often not appropriate in conditions where time is of the essence (Armstrong & Cole, 
1995). Thus, team members are increasingly reliant on emerging communications technologies 
to perform their tasks (Hollingshead & McGrath, 1995; Ilgen et al., 1993). Yet, little is known 
about the performance effectiveness of distributed collaboration. Through a review of the extant 
literature on team performance, we will identify the salient interaction processes and features of 
effective teams. The processes and features identified in the review will then be applied to 
consider the effectiveness of teams in distributed environments and finally, will be applied to 
distributed collaborative engineering teams. The ultimate goal of this research project is to 
develop a methodology for the assessment and continuous improvement of engineering team 
effectiveness in distributed collaborative environments. This review is intended to provide a 
theoretical foundation for subsequent empirical work. 

Team Performance Models 

Researchers often turn to well-developed models as frameworks in constructing 
methodologies to better understand a phenomenon. Team performance is no exception. 
Numerous studies on team effectiveness have been conducted in recent decades, and several 
models of team performance have been developed and proven useful. Although, it is beyond the 
scope of this paper to provide an exhaustive review, three models will be considered. These were 
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chosen because of their wide use in the field, their applicability to the current project, and their 
relative generalizability across multiple team settings. 

Team Effectiveness Model. Salas and his colleagues developed the Team Effectiveness 
Model (TEM) as a framework incorporating much of the team literature of the previous decade 
(Salas, Dickinson, Converse, & Tannenbaum, 1992; Tannenbaum, Beard & Salas, 1992). TEM 
has been instrumental in shaping other theoretical models in the 1990s (e.g., Dickinson & 
McIntyre, 1997; Weaver, Bowers, Salas & Cannon-Bowers, 1997; Urban, Bowers, Cannon- 
Bowers & Salas, 1995), and has received considerable empirical support (e.g., Urban, Weaver, 
Bowers & Rhodenizer, 1996; Urban, Bowers, Monday & Morgan, 1995). The model is an input- 
process-output model of team performance. See Figure 1. Simply put, four classes of inputs (i.e., 
task characteristics, work characteristics, individual characteristics and team characteristics) 
interact with one another and contribute to the performance outcome. Throughputs (e.g., team 
processes such as communication and coordination) mediate the input-output relationship over 
time. Performance of the team provides feedback to the input factors, and organization and 
situational factors (e.g., reward systems, environmental uncertainty) exert influence at all levels 
of the model. 

Team Architecture Model . Urban and her colleagues (Urban, Bowers, Cannon-Bowers, & 
Salas, 1995) developed the Team Architecture Model as a framework for possible team design 
interventions to improve teamwork processes. Team architecture refers to the system factors that 
influence team processes by inhibiting or enhancing the interactions among the individuals. 

Three important factors comprise the architecture of teams, each differentially affecting the level 
of interaction among team members. The factors are member proximity (i.e., the physical 
distance between members of a team), communication modality (i.e., the medium in which 
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communication takes place, such as verbal, face-to-face or computer-mediated, text based) and a 
team’s structure (i.e., the distribution of the team’s subtasks among the individual members). The 
authors note that this list is not exhaustive, but each of these factors is certainly important in 
addressing member interactions, which in turn influences team effectiveness. 

Task Circumolex . Teams vary greatly in the level of complexity and nature of the tasks they 
perform. McGrath (1984) conceived a model depicting a task classification scheme. The model, 
presented in Figure 2, is composed of a circle divided by four main task types occupying each of 
the quadrants. Each of the four categories is further divided into two sub-types. The main task 
types (or quadrants) are to generate, to execute, to negotiate, and to choose. The circumplex is 
further divided by an x- and y- axis. The x-axis represents a continuum on which tasks range 
from predominately cognitive to primarily behavioral in nature. The y-axis represents a 
continuum ranging from collaboration to conflict resolution. McGrath (1984; Hollingshead & 
McGrath, 1995) maintains that all variations of tasks that a team may be engaged in can be 
described in terms of the Task Circumplex. Further, Hollingshead and McGrath (1995) point to 
the lack of consistency in research in understanding the influence of technology (e.g., computer- 
mediated conferences) on team processes due to neglecting the variations in task complexity. 

Understanding Team Performance: Collaboration Effectiveness 
A team can be defined as “a distinguishable set of two or more people who interact, 
dynamically, interdependently, and adaptively toward a common and valued goal-objective- 
mission, who have each been assigned specific roles or functions to perform, and who have a 
limited life-span of membership” (Salas et al., 1992, p. 4). This definition is applicable to teams 
in many environments (e.g., command-and-control settings, aviation cockpits, and product 
development teams). The extant literature on team performance can be generalizable across 
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many types of groups working interdependently provided that key defining features of the teams 
are appropriate (Baker & Salas, 1997; also see McIntyre & Salas, 1995 for further review of the 
subject). Engineering teams often involve many specialties (e.g., mechanical, composite 
materials; Reid, Reed & Edworthy, 1999) and work with various functional divisions (e.g., 
manufacturing, marketing, finance; Hauptman & Hirji, 1996). For these reasons, engineering 
team effectiveness is highly dependent on the collaboration between the members of the team. 
Therefore it is the aim of this review to identify the processes characteristic of high performing 
teams and attempt to generalize those processes to engineering collaborative teams and 
determine which processes are applicable in distributed environments. We will also discuss the 
factors that are known to influence team processes, such as the team’s tasks, characteristics, and 
composition. 

Team Processes 

A great deal of conceptual, theoretical and empirical research has emerged in previous 
decades concerning the processes of high performing teams (see Militello et al., 1999; Paris, 
Salas, & Cannon-Bowers, 2000). As described by the Team Effectiveness Model (TEM), 
processes serve an important role as throughput variables in determining team performance 
(Salas et al., 1992; Tannenbaum et al., 1992). Considerable overlap exists in the independent 
bodies of research and certain common themes have emerged. Each of twelve separate processes 
are defined in Table 1 and discussed successively in terms of their relevance to team 
performance; any empirical support is provided. The processes listed were chosen because of 
their theoretical distinctness. Although, similarities exist, each has subtle differences. To our 
knowledge, no comprehensive empirical study (e.g., factor analysis) has been conducted to 
determine if these processes are empirically distinct. Moreover, such a study is largely unfeasible 
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due to the fact that it is difficult to obtain the large numbers of teams required to appropriately 
perform such statistical procedures. 

Mission analysis . As the operable definition of a team suggests, a goal or vision must be 
present to differentiate a group of individuals from a team. For a team to be effective, this goal 
must be clear and shared by the members (Marks, Mathieu, & Zaccaro, 2001). Further, the tasks 
of each individual must be aligned in accordance with the mission. For this to occur, regular 
attention should be placed on the status of the mission and the activities of the team (Prince & 
Salas, 1993). Stout, Salas and Carson (1994) found ratings of the pilots mission analysis (MA) to 
be significantly related to the number of targets destroyed and the overall mission performance in 
a low-fidelity flight simulation task. MA was assessed with items such as “devised long-term and 
short-term plans” and “critiqued existing plans” (Stout et al., 1994, p. 184). 

Team orientation . Although attitudes are not themselves behaviors, behaviors are certainly 
influenced by attitudes (Petty, 1995). Team orientation refers to the attitudes that members have 
towards one another and the team task (Dickinson & McIntyre, 1997). Interaction among the 
team members is likely to be lower without an attitude of task cohesion (Zaccaro, Gualtieri, & 
Minionis, 1995). Further, team norms must be mutually accepted among the members for the 
team to succeed, and members should profit from the feeling of team membership (Dickinson & 
McIntyre, 1997). Harris and Bames-Farrell (1997) found evidence for six separate processes 
significantly contributing to subjective performance appraisals: team orientation, team 
leadership, communication, monitoring, back-up behavior, and coordination. 

Resource distribution . Effectively matching expertise with task responsibilities of the team is 
key to team performance (Fleishman & Zaccaro, 1992). Each member should be utilized 
maximally according to each member’s resource contributions (Militello, Kyne, Klein, Getchell 
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& Thordsen, 1999). This implies that adjustments must be made when discrepancies exist such 
as reallocation of the sub-tasks due to some event (e.g., membership changes, task complexity 
increases/decreases, etc.). 

Communication . The exchange of information is vital to the success of two or more 
individuals working as a team (Dickinson & McIntyre, 1997). The purpose of communication is 
often to clarify misunderstandings and to acknowledge the receipt of information (e.g., 
grounding — establishment that mutual understanding has occurred between listener and speaker) 
and may not always be verbal (e.g., head nods; Reid et al., 1999). Closed-loop communication is 
a particular sequence of exchanges whereby the receiver acknowledges receipt by a return 
message, often a repeat of the initial message to covey a mutual understanding (McIntyre & 
Salas, 1995). Empirical support exists for the amount, quality and sequencing of communication 
in determining team performance (Bowers, Jentsch, Salas, & Braun, 1998; Harris & Bames- 
Farrell, 1997; Stout et al., 1994). 

Leadership . The formal authority to lead in a team may not be vested in one member 
(Dickinson & McIntyre, 1997). For example, Stewart and Barrick (2000) describe relatively 
autonomous teams as those in which the team is reasonably free of external supervision and 
characterized by self-leadership shared among team members. In addition, providing leadership 
for the team may be a responsibility that is shared among team members, even in instances 
where a formal leadership role is identified. Despite who is in charge, the process of providing 
direction and structure to others in the team is demonstrably important in high performing teams 
(Dickinson & McIntyre, 1997; McIntyre & Salas, 1995; Stout et al., 1994) and can affect other 
processes such as decentralizing communication patterns (Stewart & Barrick, 2000). 
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Coordination . Effectively synchronizing and integrating the individual task activities of a 
team is fundamental to its success (Dickinson & McIntyre, 1997; Fleishman & Zaccaro, 1992; 
Marks, et al. 2001). Reid et al. (1999) express the advantage that sketching provides as a 
coordination function for design engineers indicating that multiple means are available for 
sharing and interacting with others. Turner, Turner and Horton (1999) note the advantages of 
sketching on a whiteboard in developing new ideas as well as updating and coordinating with 
other members. Brannick and his colleagues found measures of observed team coordination to be 
highly correlated with performance ratings and communication frequencies in a low-fidelity 
flight simulation task (Brannick, Roach & Salas, 1993). This suggests that many team processes 
are interdependent (i.e., one may influenced by the presence or absence of others). 

Timing. Not only are individuals responsible for meeting deadlines in organizations, but so 
too are the teams in which they work. Because teams are comprised of individuals completing 
interdependent subtasks, effectively completing work according to prescribed timelines is 
critical. This process becomes more critical the more dependent the team members are on the 
output of other members. Sufficiently coordinating team timelines requires the pacing of both 
individual activities and the general organization of team resources (Fleishman & Zaccaro, 

1992). This also implies a team-oriented understanding of the task. Militello et al. (1999) 
maintain that teams must exercise time management and engage in team planning to be effective. 

Mutual performance monitoring. The need for team members to monitor the behaviors and 
performance of each other is a seemingly ubiquitous process in the team performance literature. 
Given that many team tasks are interdependent, it is essential that each member perform 
optimally for maximum team effectiveness (McIntyre & Salas, 1995). To compensate for 
deficiencies, constant vigilance is required (Militello et al., 1999). Therefore, it is not only 
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essential that members be individually competent in their own tasks, but also proficient in 
understanding the effectiveness of other team members’ responsibilities (Dickinson & McIntyre, 
1997). The monitoring of others’ activities makes many assumptions. For example, one must be 
able to view and recognize the performance effectiveness of those monitored (Fleishman & 
Zaccaro, 1992; Militello et al., 1999). Further, this implies that members have the motivation and 
ability to provide feedback or support when required (Dickinson & McIntyre, 1997; McIntyre & 
Salas, 1995). McIntyre and Salas (1995) note that monitoring should become an implicit contract 
among the members to prevent feelings of “spying”. 

Intra-team feedback. Provided team members are able to engage in performance monitoring, 
it is expected that they should likewise be able to provide information about the status of other 
teammates’ functioning. Feedback refers to the giving, seeking and receiving of performance 
related information among the members of a team (Dickinson & McIntyre, 1997). Members 
must wield the assertiveness and willingness to both provide and accept the criticism of others to 
perform effectively (McIntyre & Salas, 1995). There should be no obstacles to providing 
feedback of others performance such as psychological distance (e.g., rank or tenure; McIntyre & 
Salas, 1995). Brehmer and Allard (1991) provide evidence for the effects of feedback on 
decision-making tasks. They found significant increases in performance when subjects were 
provided task-related feedback in a command and control task. Rasker, Post and Schraagen 
(2000), also using a command and control task, empirically tested the effects of two types of 
feedback: during execution of a task and post-execution of a task. Teams performed 
significantly better when they were able to provide feedback during the execution of a task. 
Incidentally, they found that feedback provided between sessions of a task provided some 
performance increases, but was not as effective as the during condition. They also found 
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communication content to be different in the two feedback conditions. Feedback during the 
execution of a task consisted mostly of activity-based exchanges (e.g., communicating what each 
member is doing), while feedback between performance sessions consisted mostly of evaluation 
and determining strategy. Members providing feedback during the execution of a task seemingly 
knew what information needed to be exchanged. 

Back-up behaviors. In addition to providing feedback, team members must also be able to 
provide technical assistance when gaps and inefficiencies are noted (Dickinson & McIntyre, 

1997; McIntyre & Salas, 1995). Members must not only be willing to both provide help to 
others, but must not be reluctant to seek help when needed (McIntyre & Salas, 1995). Indeed, 
providing feedback and back-up assistance to others depends on adequate monitoring and 
proficiency in the other team members’ tasks. 

Motivational functions. Perhaps the most difficult to operationally define and measure 
(Fleishman & Zaccaro, 1992), are the functions related to developing and accepting team norms. 
However, team maintenance activities such as establishing performance objectives and 
generating task commitment are critical to team performance (Fleishman & Zaccaro, 1992). 
Motivating others to maintain high standards of performance may be accomplished through other 
processes such as feedback of individual performance or team successes (Marks et al., 2001). 
Marks and her colleagues (2001) have also observed that teams can dissuade members by the use 
of negative comments, which reduce confidence and cohesiveness. 

Cooperation. Brannick et al., (1993) defined interpersonal cooperation as “the quality of team 
member interchanges” (p. 294). They provided validity for the construct as they measured it and 
demonstrated its role in performance effectiveness. Cooperation can be viewed behaviorally as 
conflict management or providing encouragement (Brannick et al., 1993; Militello et al., 1999). 
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The interpersonal processes labeled by Marks and her colleagues includes the importance of 
conflict and affect management of the members (2001). Effective teams should strive for 
harmony (Militello et al., 1999). 

Task, Team, and Individual Characteristics 

Each of these twelve processes described above interdependently contribute to the 
effectiveness of teams performing complex tasks. However, team research in the previous decade 
has supported the Team Effectiveness Model in its assertion that certain inputs exert influence on 
the need and impact of the processes in teamwork. For instance, processes will have different 
effects on the performance outcome depending on task characteristics, team characteristics and 
individual member characteristics. We review each of these influences in the ensuing sections. 
Task Related Influences 

Several models of team performance incorporate the nature of the task into their frameworks 
(see Guzzo & Shea, 1992; Salas et al., 1992). The task demands of a team and its members have 
many documented effects. For instance, the Team Effectiveness Model (TEM; Salas et al., 1992; 
Tannenbaum et al., 1992) indicates that the outcome or effectiveness of a team is affected by the 
inputs of task complexity, task organization and the task type. The work structure has also been 
demonstrated to influence a team’s effectiveness (Salas et al., 1992; Stewart & Barrick, 2000). 
Further, Guzzo and Shea (1992) in a review noted that tasks affect performance in at least three 
ways: via member motivation, moderating member interaction and effectiveness, and as 
determinants of the requirements and interactions among the members. 

Task type. The type of task that a team performs has been shown to moderate the effects of 
intrateam processes and performance (Stewart & Barrick, 2000). Using McGrath’s (1984) task 
typology, Stewart and Barrick found that the performance of teams engaged primarily in 
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behavioral tasks was relatively unaffected by intrateam processes, whereas the performance of 
teams engaged primarily in conceptual tasks was significantly related to the team’s processes. 
Steiner (1972) developed a typology that describes team tasks as either disjunctive (i.e., one 
exceptional member of the team can perform the task), conjunctive (i.e., performance depends on 
all team members’ contributions), additive (i.e., performance is dependent on the summation of 
the members’ effort), or discretionary (i.e., resources can be combined in any way). As such, 
task demands are moderators of member interaction and overall team effectiveness. For example, 
in a disjunctive task, little team process or member interaction will be needed to complete the 
task because it is likely that the task will be performed by the “best” member of the team. 
However, in a conjunctive task, interaction among the members will have a much greater 
influence on the team’s effectiveness. Not only does the type of task influence team 
effectiveness, but so does the method for performing the tasks. There are multiple methods for 
performing some tasks, and these methods have varying effects on performance outcomes (Sauer 
et al., 2000). 

Task complexity. Task complexity which refers to the demand characteristics of the subtasks 
includes variables such as time pressures/demands, amount of workload, level of information 
processing needed, number of dimensions a task has, and the degree to which those dimensions 
are prone to change (Maznevski & Chudoba, 2000; Salas et al. 1992). Many studies have 
demonstrated the influence of task complexity on team performance (Urban et al., 1996; Urban, 
Bowers, Monday, & Morgan, 1995; Zaccaro et al., 1995), and the relationship of task complexity 
with task organization. That is, greater complexity in the subtasks of a team necessitates greater 
interdependence among the members. Task organization refers to the interdependencies that exist 
among the subtasks (Salas et al., 1992). Interdependence of team members can best be described 
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by Thompson’s (1967) typology. Interdependence can be pooled (i.e., little or no interaction is 
necessary), sequential (i.e., members are dependent on those in previous steps along a chain), 
reciprocal (i.e., direct two-way interactions are necessary), or comprehensive (i.e., a complex 
network of dependence exists among members). Related to the task organization or 
interdependencies is the work structure of a team (i.e., the manner in which the tasks are 
distributed among the members; Stewart & Barrick, 2000; Urban, Bowers, Cannon-Bowers, & 
Salas, 1995). The role of team structure will be discussed under team characteristics below. 

Motivation. Guzzo and Shea (1992) citing Hackman and Oldham (1980) refer to the 
motivational characteristics of tasks. Various tasks will elicit varying levels of effort and 
therefore affect team performance to the extent that effort is related to performance. Weaver, 
Bowers, Salas, and Cannon-Bowers (1997) note that team performance is a function of both 
taskwork input as well as teamwork input, and there is a complex interdependence between the 
motivation to perform taskwork and the motivation to perform teamwork. Therefore, to fully 
understand team effectiveness, one must account for the role of tasks in eliciting motivation and 
the relationship of taskwork motivation to teamwork motivation. Ultimately, to effectively 
understand the relationship of teamwork processes to team performance, one must account for 
the nature of the tasks performed. To do so, a team task analysis is a prerequisite to the 
measurement of team performance (Paris et al., 2000) 

Temporal changes . Having discussed the effects of variations in task type and complexity on 
team processes, we must address the temporal issues of tasks. Ancona and Caldwell (1990) 
working with new product development teams identified a series of three stages that teams 
progress through. The teams’ predominant task activities were contrasted greatly between the 
three stages. They noted that product development teams have a creation phase where the 
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predominant activity is exploration of their own resources to discover what the team has 
available to them. In the second stage, development, the predominant task activity is exploitation 
of the resources that the team has acquired. Finally, in the third stage, diffusion and ending, the 
dominant activity is exportation of the product to others in the organization. Although the greater 
task remains the same throughout the cycle (i.e., create a new product), the interpersonal 
processes and intergroup relations change greatly throughout the three phases. 

McGrath (1990) developed a similar temporal model of the stages and functions of teams. He 
contended that teams go through each of four stages (i.e., inception or goal choices, problem 
solving or means choices, conflict resolution or political choices, and execution or goal 
attainment) at each of three functional levels (i.e., production, member support, and group well- 
being). Different skills and processes are required at each stage and for each functional activity. 
More recently, Marks et al. (2001) proposed that teams cycle in a rhythm of task 
accomplishment. Different team processes are required of teams throughout the cycle. For 
instance, Marks and her colleagues argued that mission analysis is predominant during transition 
periods, and that coordination and monitoring are more essential in the action periods of the 
cycle. They also proposed that certain interpersonal processes, such as motivational functions 
and conflict management are essential throughout the team’s performance cycle. 

In yet another temporal model of design, Turner et al. (1999) describe three dimensions of 
transformation. In the first dimension, distributed cognition, knowledge is distributed among the 
team members and must become centralized. Secondly, the design stage moves from one that is 
provisional to a more permanent mutually agreed upon stage. Finally, the object of design must 
cross team boundaries and be shared with those external to the team. Turner and his colleagues 
believed that each of these transformations can be facilitated by emerging technologies. 
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Team Characteristics 

The focus of the unit of analysis of team performance has shifted in recent years from the 
aggregate of individual performances to that of the team as a whole (McIntyre & Salas, 1995). 

As such, teams have unique characteristics that contribute to team performance (Salas et al., 
1992). For instance, the team performance model (Nieva, Fleishman & Reich, 1978) suggests 
that such team characteristics as size, group cohesiveness, intra- and inter-team cooperation, and 
power distribution affect team performance. As already noted, the task of the team can 
significantly affect the team’s structure, which in turn has been shown to affect team 
performance in various ways (Stewart & Barrick, 2000; Urban et al., 1996). Another team 
characteristic that has developed theoretically in the past decade is that of team knowledge. Paris 
et al. (2000) acknowledge that the advent of the shared mental model is in fact the “hallmark of 
the nineties” regarding team research. Our focus will be on the team characteristics, structure and 
knowledge. 

Team structure. The structure of a team refers to the distribution of the subtasks, 
responsibilities, and authority among the individual members of the team (Salas et al., 1992; 
Stewart & Barrick, 2000). Subtasks may be distributed in a manner that several members 
perform the same task (i.e., redundancy) to ensure adequate performance (Salas et al., 1992), or 
the subtasks may be distributed such that individual members perform independent subtasks 
given their various expertise (Hollenbeck et al., 1995). Further, teams may be differentiated in 
their structure by a hierarchical distribution (i.e., team members hold unique expertise and are 
subject to a formal chain-of-command) or non-hierarchical distribution (i.e., individual team 
members are non-specialized and share common information and capabilities; Urban, Bowers, 
Monday & Morgan, 1995). Indeed, the task largely determines the structure of a successful team. 
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It may not be feasible for some complex tasks to be performed by many members of the same 
team, nor can all members of a team possibly possess all the needed knowledge to perform the 
task in its entirety. This is perhaps the initial appeal of using teams in organizations in the first 
place. 

Structure is an important factor influencing team processes. The effective team structure will 
alleviate the workload burden of a team’s members by allowing the members to share 
responsibilities (Salas et al., 1992). Several studies have found an overall performance 
advantage in nonhierarchical structures (see Urban, Bowers, Monday & Morgan, 1995), albeit 
nonhierarchical structures are not always practical. Further, the structure of a team can affect the 
communication structure as well as other coordination processes that exist within the team 
(Stewart & Barrick, 2000; Urban, et al, 1996; Urban, Bowers, Monday & Morgan, 1995). For 
instance, a team structure that encourages self-leadership or autonomy may induce a 
decentralized communication structure that will in turn affect performance. This effect, as 
previously noted, will be moderated by the task type (e.g., self-leadership is associated with 
higher performance in conceptual tasks - but, self-leadership is associated with diminished 
performance on behavioral tasks; Stewart & Barrick, 2000). 

Team knowledge. Team knowledge refers to the collection of task- and team-relevant 
knowledge held by the team members and their collective understanding of the current situation 
(Cooke, Salas, Cannon-Bowers, & Stout, 2000). According to Cooke and her colleagues, team 
knowledge is a component of the greater construct team cognition, which encompasses such 
phenomena as team decision-making, team vigilance, team situation awareness and team 
knowledge. Team knowledge also contains two subsets: team mental model (TMM) and team 
situation model (TSM). The TMM can be defined as the collective task- and team-relevant 
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knowledge that the team members bring to a situation. TMM is generally long-lasting, acquired 
through training or experience and exists prior to specific tasks. TSM is defined as the collective, 
dynamic understanding of a specific situation. In contrast to TMM, TSM is more fleeting and 
dynamic, situation specific and acquired during the completion of a task. Research has provided 
evidence that team performance is maximized when knowledge is accurate and appropriately 
distributed among the team members such that appropriate strategies can be utilized to cope with 
the team tasks (for review see Cooke et al., 2000; Paris et al., 2000). Shared team knowledge is 
also thought to enable implicit coordination and increase communication efficiency among team 
members when explicit communication/coordination is hindered (Cooke et al., 2000; Rasker et 
al., 2000; Marks, Mathieu, & Zaccaro, 2000; Stout, Cannon-Bowers, Salas, Milanovich, 1999). 
Shared knowledge also contributes to vision clarity and stability (Lynn & Reilly, 2001; Lynn, 
Reilly & Akgiin, 2000), which could lead to better performance. 

Several recent studies have demonstrated methods for fostering team knowledge and 
addressed the effects of team knowledge on team performance. Stout and her colleagues (1999) 
demonstrated that prior planning enhanced the shared mental models of teammates, which 
resulted in more efficient communication and information passing in advance of explicit 
requests, enhancing performance in high-work load situations. Similar to planning is the role of 
leader briefings. Marks et al. (2000) demonstrated that pre-mission leader briefings enabled the 
development of mental models that in turn affected communication processes and flexibility in 
novel settings. They further demonstrated that knowledge similarity was more important for 
team effectiveness than initial knowledge accuracy. Another study demonstrating the link 
between the TMM and team processes was conducted by Rasker et al. (2000). This study showed 
the effects of two-types of intrateam feedback on TMM - performance monitoring (i.e., feedback 
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during the activity) and team self-correction (i.e., feedback between performance sessions). 
Teams engaged in performance monitoring were much better informed of the situation and better 
able to adapt to the team task. Self-correction was shown to be better than no feedback, but not 
as effective as the ongoing communication of performance monitoring that allowed the 
continuous updating of the TMM. In an attempt to better understand the knowledge management 
of effective new product development teams, Lynn and colleagues (Lynn & Reilly, 2001; Lynn 
et al., 2000) noted that high performing teams have strategies for recording, reviewing and filing 
of project information in a manner that can be later retrieved. This assists in maintaining the 
team’s vision clarity and leads to vision stability. 

Individual Team Member Characteristics 

To this point, we have maintained the focus on the team in the aggregate, the team processes, 
the role of the team task and the characteristics of the team. However, no understanding of team 
performance would be complete without addressing that which composes the team - the 
individuals. Individual team members bring many attributes to the team setting. First, team 
members must have the knowledge, skills and abilities (KSAs) to perform their technical tasks 
effectively. Aside from the technical KSAs however, individuals contribute in their dispositions 
(e.g., personality traits, general cognitive abilities, etc.) and their teamwork KSAs (LePine, 
Hollenbeck, Ilgen & Hedlund, 1997; Neuman & Wright, 1999; Stevens & Campion, 1994; 
1999). 

Dispositions. A considerable number of studies have shown the relationship of individual 
task performance with relatively stable characteristics such as general cognitive ability (g) and 
personality traits (see LePine et al., 1997). However, only recently have studies begun to view 
the relationship of such traits with team performance. For example, g has been shown to be 
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related to individual performance measures, and in at least two studies using a conjunctive model 
(i.e., performance is dependent on the weakest link) g was shown to be modestly related to team 
performance (LePine et al., 1997; Neuman & Wright, 1999). Low leader g was found to stifle the 
effects of high g in the team members in teams with low horizontal substitutability (i.e., little 
redundancy of tasks; LePine et al., 1997). However, LePine et al. found that even when 
horizontal substitutability was low, teams with low g (i.e., a member with low g) were able to 
compensate for the team deficiency by altering communication patterns. Having one or two 
members high in g resulted in anticipating the communication needs of those low in g. This 
compensatory behavior was not found for teams low in conscientiousness. 

Conscientiousness is a relatively stable personality trait with the sub-categories of 
competency, dutifulness, achievement, and self-discipline (Neuman & Wright, 1999). In a study 
of teams of human resource workers, Neuman and Wright (1999) found conscientiousness to be 
significantly related to both task performance and work accuracy above that which g and 
individual technical KSAs would have predicted. Agreeableness was also found to be positively 
related to task performance. Agreeableness is a personality trait representing general quality of 
interpersonal interactions consisting of the personality facets trust, straightforwardness, altruistic, 
compliance, modesty, and tender-mindedness (Neuman & Wright, 1999). This personality trait 
was also significantly positively correlated with measures of interpersonal skills. In addition to g, 
conscientiousness, and agreeableness, other research has demonstrated that assertiveness in the 
team members is essential to team performance. For instance, team members must show a 
willingness to provide backup behaviors when needed (McIntyre & Salas, 1995) and make 
suggestions when appropriate (Stout et al., 1994; Prince & Salas, 1993). 
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Teamwork KSAs. Previous work by Stevens and Campion (1999, 1994) identified and 
validated two main dimensions of teamwork KSAs: interpersonal KSAs and self-management 
KSAs. They investigated these KSAs because (1) they could be trainable, unlike dispositions 
and cognitive abilities, and (2) they could be used in personnel selection. The interpersonal 
KSAs include sub-categories of conflict resolution, collaborative problem solving and 
communication KSAs. Each of these sub-categories includes specific knowledge, skills and 
abilities needed for optimum team effectiveness (e.g., recognize obstacles to collaborative group 
problem solving and implement appropriate corrective actions). Seat and Lord (1998) identified 
similar “soft skills” as necessary components of interaction among engineering problem solvers 
and developed a training initiative that incorporated such skills as interviewing, questioning, 
exchanging ideas, and managing conflict. In addition to these interpersonal KSAs, several 
studies have cited the adaptability and flexibility of the team members to cope with situational 
and task demands as an important interpersonal KSA (Marks et al, 2000; Stewart & Barrick, 
2000; McIntyre & Salas, 1995; Stout et al, 1994; Prince & Salas, 1993). 

The second main dimension of teamwork KSAs identified by Stevens and Campion (1999, 
1994) are the self-management KSAs. Subcategories of the self-management dimension include 
goal setting and performance management, and planning and task coordination. Again, each of 
these sub-categories has specific KSAs associated with effective team performance (e.g., to 
monitor, evaluate, and provide feedback on both overall team performance and individual team 
member performance). This latter example of a specific KSA implies that the members have 
adequate inter-positional knowledge (i.e., the knowledge of the roles of other members of the 
team: Urban, Bowers, Monday & Morgan, 1995). The absence of this knowledge, called inter- 
positional uncertainty (IPU), has had negative effects on team processes and performance 
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(Urban, Bowers, Monday & Morgan, 1995). In addition to these KSAs, team members must 
exhibit the willingness or motivation to perform the needed teamwork behaviors (Weaver et al., 
1997; McIntyre & Salas, 1995). 

Distributed Collaboration 

Recent technological advances have enabled teams to work in a distributed fashion. This has 
increased economic competitiveness in organizations for many reasons (see Ilgen et al., 1993). 
Collaboration has been defined as two or more people working together for common work 
related outcomes (Horrocks, Rahmati & Robbins-Jones, 1999). We extend to this the notion of 
distributed collaboration to mean working on a common task while separated by distance. 
Whether collocated or distributed, members of teams must perform the tasks required of the 
team. Both the technical tasks and the above mentioned teamwork tasks (i.e., team processes) 
must be fulfilled. However difficult it is to learn and exhibit teamwork processes in a collocated 
environment, teams that are distributed must overcome those challenges as well as dealing with 
emerging technologies (e.g., communication, shared workspace, and shared library technologies; 
May & Carter, 2001), and other issues involving non-face-to-face interactions (e.g., spontaneous 
encounters, modeling of mentor behavior, etc.; Armstrong & Cole, 1995). Horrocks et al. (1999) 
identified many potential reasons for collaboration, some of which are not captured in common 
notions of tasks. Their framework proposes that teams meet for externalities (i.e., ceremony, 
statutory), problem solving (i.e., identification, idea generation, etc), and support processes (i.e., 
coordination, socialization, motivation, etc.). 

Collaboration can be viewed in terms of a hierarchy of activities. Relying on Activity 
Theory, 2 Bardram (1998) illustrated three levels of collaboration: co-ordinated, co-operative, and 
co-constructive. Co-ordinated collaboration refers to the normal routine interactions of 
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individuals working from the point of view of the individual. Co-operative collaboration requires 
more interaction among the individuals working interdependently. Co-constructive collaboration 
refers to the collective re-conceptualization of the task objective. Each varies in the level of 
interdependencies and levels of awareness required of the participants. Therefore, tools aimed at 
supporting collaboration should be dynamic to the extent that the collaborative activities range in 
the hierarchy. 

Computer Supported Cooperative Work 

Computer supported cooperative work (CSCW) is an emerging body of research on the use 
of technology to support and enhance communication and other organizational activities (Olson 
& Olson, 1997). CSCW can typically be divided into four main categories of support systems: 
communication; shared workspace & mutual awareness; shared information & information 
management; and group activity support (May & Carter, 2001), and should be studied in the 
aggregate (Olson & Teasley, 1996). Because technology is growing at an exponential pace, the 
research on the effects of each of these support mechanisms is somewhat lacking. However, we 
can make limited conclusions about the effects of some technologies and communication 
(Hollingshead & McGrath, 1995). Which technology should be implemented and utilized 
depends largely on the task (e.g., generating ideas or executing performance tasks) (Hollingshead 
& McGrath, 1995), stage of team task (e.g., problem solving or execution stages; Ancona & 
Caldwell, 1990; May & Carter, 2001 ; McGrath, 1990), and the level of media richness required 
of the task (Maznevski & Chudoba, 2000). Further, ease of use should be addressed when 
implementing such support systems; technology does not help when the tools themselves further 
burden the already busy team members (Jude- York, 1998). In fact, some teams can be disbursed 
not by geographic distances, but by their individual workloads (Jude-York, 1998). Olson and 
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Teasley (1996) cite the need for management to support the use of new technologies 
implemented in any organization. They found teams to vary their level of task interdependencies 
based on the barriers the members encountered from various technologies. This implies that if 
the employees meet challenges without support of the tool use from management, the task 
interdependencies envisioned will suffer. 

Hollingshead and McGrath (1995) conducted an extensive review of the computer-mediated 
communication (CM) literature and made several conclusions. 1 Overall, CM generates less 
communication units and each unit is filled with less socio-emotional content resulting in the 
filtering of extraneous information. There is also an apparent equalization of the participation in 
the communication patterns (i.e., less participation from verbose members). Messages tend to 
become uninhibited, a phenomenon known as “flaming,” when perceptions of conflict arise 
(presumably due to lost socio-emotional content and relative anonymity). CM has also been 
viewed (depending on the task) as a more productive mode of communication (Sauer et al., 

2000 ). 

Although few studies have taken integrated approaches (i.e., viewing the effects of combined 
technologies), some have addressed differences between face-to-face interactions and the use of 
multimedia technologies (MM: includes, video, audio, text, and graphics for example). Sauer and 
his colleagues (2000) for example found that teams using MM performed remarkably similar to 
teams operating face-to-face in a knowledge acquisition task. The teams functioning with MM 
took longer on average than teams operating either face-to-face or in CM conditions, but results 
were dependent on the method for engaging in the task (i.e., structured interviewing was poorest 
in the MM condition, whereas a network technique of knowledge acquisition was superior in the 
MM condition). By viewing collocated engineering design teams, Reid et al. (1999) determined 
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that a shared workspace for visual representations, audio for communication, and a means to 
point were sufficient for effective collaboration. In the engineering design teams, a visual contact 
of the speaker was only needed for grounding (i.e., ensuring that the listeners understand what is 
being said), which could be done by a pointing device on the visual display. May and Carter 
(2001) note that static images are found to be preferred over poor quality video in automotive 
product design teams. Video was primarily used for initial meetings and was later placed in the 
background of the workspace. In fact, Reid et al. (1999) suggest that only a “small proportion of 
the activities of design teams requires the processing power and advanced display technologies 
currently under construction. . ”(p. 255). However, Brannick, et al. (1993) observed that remote 
evaluators rating audio transcripts viewed team processes differently than did on-site visual 
observers of collocated teams. This indicates that some visual, non-verbal interaction influences 
ratings of team processes. 

Team Process Barriers in Distributed Environments 

Technology issues aside, there are several difficulties that have been observed in distributed 
teams that should be addressed to enhance performance. Armstrong and Cole (1995) reported on 
their observations of teams residing in a fortune 100 company during the course of their 
interviews and consultations with the firm. We will report on four main areas of concern for 
distributed teams: spontaneous encounters, modeling behavior, out-of-sight issues, and, 
recognizing and resolving conflicts. Collocated team members have an immediate advantage 
over distributed members in that chance encounters are able to occur which results in informal 
discussion, feedback, and decision-making. These spontaneous encounters greatly reduce 
misunderstandings, and shorten the time spent in formal meetings resolving the more mundane 
issues. Recent approaches to technology development may be able to address this issue with 
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applications such as an electronic virtual “foyer” (Benford, Brown, Reynard & Greenhalgh, 
1996). A natural foyer serves many purposes for an organization such as the public face of the 
building, entranceways to the organization, and public meeting places. An electronic foyer 
should serve the same purposes. Armstrong and Cole also note the value of modeling to the 
newcomer of an organization. Both newcomers and subordinates alike are at a significant 
disadvantage if they are not able to model their mentor’s behavior. In addition, distributed 
members are often left out of important discussions, even when all members are engaged in a 
formal team meeting. Apparently, collocated cultures emerge and dominant members tend to 
forget to include the distributed members in remote conferencing. Finally, conflicts are often 
unrecognized and therefore go unresolved for lengthy periods. When managers can see the 
problem (i.e., when subordinates are collocated), the problems can be resolved quickly, but 
managing distributed members is no easy task. Unresolved conflicts lead to more intense future 
encounters and feelings of distance from the team’s core. In her study of three teams utilizing the 
same technologies, Jude- York (1998) demonstrated that CSCW systems can provide the 
technological support, and the team could still fail. This indicates the importance of social and 
process issues in teamwork. 

Potential Benefits of Distributed Environments 

We have demonstrated the many pitfalls associated with distributed collaboration such as 
technological challenges and social issues, but there are perhaps as many compelling reasons to 
further explore collaborating in distributed environments as there are disadvantages. For 
instance, there are practical and strategic competitive advantages for enabling distributed 
collaboration (see Ilgen et al., 1993). Further, using the various technological support systems 
can have individual advantages such as filtering extraneous information (e.g., Hedlund, Ilgen, & 
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Hollenbeck, 1998) and enabling asynchronous work. Several studies have proposed that effective 
distributed teams make use of such advantages as the appropriate technologies for 
communication and knowledge sharing (May & Carter, 2001; Maznevski & Chudoba, 2000; 
Sauer et al., 2000) 

The practical advantages for distributed collaboration are many. First, it may not be feasible 
for members of a distributed team to be physically present for certain phases of a task (e.g., team 
members who are also members of other teams may not be in two geographically separated 
locations in the same time period). Entire days are often lost in travel for a meeting that could 
have occurred via some technologically supported means. Travel costs are also a consideration in 
determining economic competitiveness. It has been estimated that geographically distributed 
product design teams making use of technology have reduced the time to market by as much as 
50% (May & Carter, 2001; Jude-York, 1998), which can reduce costs by millions of dollars and 
increase sales volume by billions of dollars in the automotive industry. 

Technology not only alleviates the problems of geographic differences, but also may enhance 
some communication by filtering extraneous info and allowing messages to be transferred 
asynchronously. For example, in decision-making tasks, Hedlund and her colleagues (1998) 
found hierarchical sensitivity (i.e., the extent to which the team’s leader appropriately weighs 
members’ recommendations in arriving at his or her decision for the team) to be greater in CM 
teams than in teams operating face-to-face. However, they found team informity (i.e., the extent 
to which all information potentially available to the team is actually acquired by those staff 
members who need it) and staff validity (i.e., the degree to which staff members’ 
recommendations are predictive of the correct team decision) to be higher in face-to-face teams. 3 
This indicates that the same communication mode may both enhance and inhibit factors leading 
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to “correct” decision-making. Others have noted the benefits of the relative anonymity afforded 
by the use of computer-mediated support systems (Bikson & Eveland, 1996). For instance, 
meeting anonymously obscures status differences, which enhances the input of members 
performing a divergent task (i.e., generating ideas, plans etc.). This effect is not as positive in 
conducting convergent tasks however (i.e., not all members have an equal voice in decision- 
making, voting etc.). 

When assessing the benefits of CSCW, one should always be aware of the levels of analysis 
(e.g., individual, team, organization, industry) because what may benefit the team may be an 
inconvenience to the member and vice-versa (Olson & Olson, 1997). For example, team 
members are often inconvenienced by the implementation and the maintenance required of 
knowledge management systems (Jude-York, 1998). Gutwin and Greenberg (1998) also point to 
the issues concerning control of shared work objects. This is to say, designers of shared 
workspaces should keep in mind the level of control — should each individual or the group have 
control of workspace navigation, artifact manipulation, and view representation? The team task 
dictates the response to this dilemma. 

Effective distributed collaboration requires that the task be matched with the appropriate 
technology available (Maznevski & Chudoba, 2000; Sauer et al., 2000). This is to say that the 
media richness required of the task should be matched with the media richness available by the 
supporting technology. Maznevski and Chudoba (2000) propose that for effective teams, task 
complexity will dictate the number, mode, and complexity of communication incidents. Future 
work on asynchronous communication should address the problems associated with 
redundancies and member awareness. For example, when team members collaborate via e-mail, 
members often forget the context of each message requiring the reader to have to re-read 



Collaboration Effectiveness 30 


previous messages. (Neuwirth, Morris, Regli, Chandhok & Wenger, 1998). Neuwirth et al. 
demonstrated the benefits of using task-tailorable messaging systems to greatly reduce 
redundancies and enhance member awareness. Technologies such as information libraries are 
also allowing teams to reduce redundancies and focus on problem-solving and decision making 
in team meetings rather than being bogged down by mundane issues of bringing the other 
members up to date; this also allows for part-time teammates to stay abreast of the team 
functions (Jude- York, 1998; Lynn & Reilly, 2001; Lynn et al., 2000). Maznevski and Chudoba 
also suggest that distributed teams should develop a rhythmic pattern of regularly scheduled 
(preferably face-to-face) encounters. The distance between these encounters can be increased to 
a year or more if the members share a common view of the task, have strong relationships and 
commitment to the team, and regular meetings utilizing a less rich medium are conducted. The 
effects of spatial distances and technological advancements on team functioning and 
performance are summarized in Table 2. 

Engineering Tasks 

The ultimate purpose of this review is to provide a foundation for understanding the critical 
team processes that contribute to the effectiveness of collaborative engineering teams in 
distributed environments, so that a methodology for the continued assessment and improvement 
of the overall effectiveness of such teams can be developed. For collaborative engineering teams, 
and indeed for all teams, a team task analysis is critical to assessing and improving team 
performance (Paris et al., 2000; Salas et al., 1992). Team task analysis requires an adequate 
understanding of the individual subtasks that the team performs and an examination of the 
pertinent teamwork processes that are needed for the success of the team. This is to say, we need 
to know what engineering teams do and how they do it. Engineers typically are engaged in the 
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design, re-design, or assessing the feasibility of products (May & Carter, 2001; Reid et al., 1999; 
Sauer et al., 2000). In designing products, methods for acquiring the expert knowledge of others 
are highly useful for the engineering team (Sauer et al, 2000; Seat & Lord, 1998). Due to 
increasing task complexities and competitive market conditions, teams of specialists are often 
employed for the creation, development, and diffusion of products (Ancona & Caldwell, 1990). 
Teams of specialists often are comprised of engineers from various domains such as mechanical 
engineering, composite materials engineering, and engineering systems among others (Reid et 
al., 1999), which work along side functional specialists from different organizational areas (e.g., 
manufacturing, marketing, finance; Hauptman & Hirji, 1996; May & Carter, 2001; Sauer et al., 
2000 ). 

Through a process known as concurrent engineering (CE), cross-functional teams are able to 
simultaneously design, manufacture, test and support various products greatly enhancing the 
firm’s economic competitiveness (Hauptman & Hirji, 1996). In CE, designers are often required 
to maintain a collaborative relationship with suppliers, manufacturers, distributors and the 
corporate center (May & Carter, 2001). These relationships demand adequate team processes for 
effective performance. Further, geographic challenges often lead to the need for distributed 
collaboration. Hauptman and Hirji (1996) suggest that information flow is significantly related 
to meeting project budget and temporal goals in CE. Lynn and his colleagues (Lynn & Reilly, 
2001; Lynn et al., 2000) recommend that new product development teams establish a knowledge 
management sequence that involves the adequate recording and filing of the team’s knowledge 
so that future reviewing is possible. Sufficient knowledge management has been demonstrated to 
contribute to increasing vision clarity and stability among the staff and to decrease the time to 
market for new products (Lynn & Reilly, 2001; Lynn et al., 2000; May & Carter, 2001). The 



Collaboration Effectiveness 32 


above is only a cursory description of the tasks that engineers perform. As the field of 
engineering is vast, so too are the tasks relevant to engineering collaboration. However, 

McGrath (1984) has provided a common framework with which to categorize tasks that 
engineers in specific teams perform. Using the above review, and an adequate task analysis, a 
model of effective engineering collaboration will be developed. 

Summary 

Through our review of the literature on team performance we have identified twelve 
theoretically distinct functions that we believe are salient to the effectiveness of teams in 
generalized settings; these are presented in Table 1. Several relevant factors were discussed that 
should be addressed when mapping the team processes mentioned above to any specific team. 
Therefore, we noted that the processes are relevant to all teams; however, their importance in 
individually determining performance depends largely on factors such as task characteristics, 
team characteristics, individual influences, and the like. 

We also determined that distributed teams have unique issues that technology can both 
encumber and enhance. These factors are summarized in Table 2. Although future research 
should explore each issue and technological advancement in contributing to team performance, 
we have provided a cursory summary of the research to date. We intend to use this review as a 
prelude to the development of a model of effective distributed team performance. Coupled with 
an adequate task analysis of engineering collaborative teams, we will develop a methodology for 
the assessment and continued improvement of distributed collaborative engineers. Such a 
methodology will be useful in providing a framework for the advancement of future 
collaborative technology and distributed team effectiveness. 
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Footnotes 

1 CM in this context generally refers to text-based transfers of information. 

2 For a review of activity theory see Bardram (1998). 

3 The interested reader is directed to Hollenbeck, et al. (1995) for a complete discussion of the 
effects of these factors on decision-making. 
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Table 2. 


Summary of distributed collaboration effects. 


Issue 

Costs of issue 

Benefits of issue 

Potential Technology? 

Less socio- 
emotional 
content in CM 

Increase tendency for 
“flaming” 

Staff validity & 

Team informity 
decreased 

Extraneous info filtered 
Hierarchical sensitivity 
increased 

Reduce psychological 
distance 

Make use of other tools 
when appropriate 
MM - make use of 
many tools 

Less 

Reduction in 

Equalization of 


communication 
units in CM 

loquacious members 

participation 


Poor quality of 
video 

Used for “grounding” 
Visual non-verbal 
affects ratings of 
processes 

Video not as necessary as 
thought; static images 
preferred 

Assess what is needed 
Don’t use video 
Improve video 

No spontaneous 
encounters 

No informal 
-discussion 
-feedback or 
-decision-making 

Could this also be more 
productive? 

Virtual work spaces 
(coffee pot on-line) 
Virtual “foyer” 

No modeling 

-Slower more tense 
socialization 
-Difficult to 
transition to in-group 
member 

Are most engineers better 
at working alone? Are they 
not already skilled? 

Newcomers collocated 
with experienced 
personnel 

“Out-of-sight” 

Team suffers when 
all members are not 
engaged 

Could also be seen as 
filtering 

e-assertiveness 
technology can 
increase awareness of 
quiet distributed 
members 

Unrecognized 

conflicts 

Go unchecked until 
boiling point. 
Disrupt team 
harmony 

(Should note often due to 
misunderstandings) 

Improve overall 
processes & 
relationships 
i.e., teambuilding & 
communication 

Need common 
view of task 

W/o shared view task 
is difficult even when 
collocated 

With shared view, less 
interaction is needed 

Information/knowledge 

warehouse 

Travel 

Fewer face-to-face 
meetings 

Time is spared 
Travel costs reduced 

Proper “tool bag” 
reduces the need 


Note. CM = computer-mediated communication; MM — Multimedia communication (e.g., text, 


video, and audio); See text for descriptions of Hierarchical sensitivity. Staff validity and Team 
inform ity. 
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Figure Captions 

Figure 1. The Team Effectiveness Model (TEM). Adapted from Tannenbaum, Beard and 
Salas (1992). 

Figure 2. The Task Circumplex. Adapted from Hollingshead & McGrath (1995). 



anizational & Situational Characteristics 



Figure 1. 
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Team Task Analysis of IS AT - Inter-center Systems Analysis Team 



Team Task Analysis of ISAT - Inter-center Systems Analysis Team 
During the August 2001 Workshop as viewed from LaRC 
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ISAT - Overview 


The Inter-center Systems Assessment Team (ISAT) is charged with performing advanced technology assessments 
on concept vehicles. The technology evaluations are used by the 2 nd Gen. program to make technology investment 
strategy decisions. The team has previously conducted these analyses in the Collaborative Engineering Center 
(CEC) at MSFC. Since the analysts that comprise the team are located at various NASA facilities (e.g., LaRC, ARC, 
JSC — see Figure 1), a goal exists to work in a distributed environment. Currently, the team meets for a week in the 
CEC to perform 15-16 analyses — approximately 4 per day. The task is largely sequential in nature meaning that the 
work flows unidirectionally from a given starting point. The ultimate goal of the ISAT team is to increase efficiency 
and work from the individual analysts workstation without the need for travel to the CEC at MSFC. The succeeding 
task analysis is based on documents and presentations obtained from RSTS, various informal interviews of RSTS 
members and ISAT members as well as observations of the activities that occurred during the weeklong workshop in 
August 2001 as observed from the LaRC location (activities occurred in various locales). 

ISAT - Team Tasks 


ISAT is comprised of various sub-teams representing various disciplines. Currently, the disciplines include vehicle 
closure (which includes weights and sizing experts that work with trajectory experts), safety and reliability, 
operations, and costs & economics. Each discipline has various analysis tools to perform the particular evaluations. 

A goal in the future is to include more discipline assessments for each concept vehicle to aid in the vehicle 
recommendations. Additional disciplines include structures & thermal, propulsion system, aerodynamics & control, 
avionics & power, and IVHM. The team recently performed a workshop in a semi-distributed environment, 
illustrated in Figure 1 . 

The flow of work is illustrated in Figure 2. Prior to these assessments, several activities have occurred, 1) the 
reference vehicle has been defined and validated, 2) the technology data have been collected, and 3) the technology 
data have been used to modify the reference vehicle models. Having these steps completed, the assessments may 
commence. Each step will be described in turn below. In the current situation using ADTT as a database 
management tool, between 1 and 3 different models are used per vehicle (i.e., HAVOC, INTROS, and CONSIZ). A 
decision is made at some point (it is not clear if this is a post hoc or a priori decision) as to which model to base the 
recommendations on. Each of these models is briefly discussed below. In diagram in Figure 2, the rows represent the 
various model assessments and the columns represent the various disciplines (e.g., weights & sizing, safety, 
economics, etc). The rows may be run concurrently, however the columns must be conducted sequentially. That is, 
the various models are independent of one another, but within the models, the flow of work is sequential. During the 
workshop, ISAT ran assessments using all three models for two vehicle cases and only the INTROS model for the 
final case. Each discipline is comprised of one or more analysts depending on the needs of the assessment. 


Team Structure 


The overall interdependence of the team task can be described as sequential, however as will be described below, 
the various sub-teams operate in a reciprocal manner. The overall team task is conjunctive (i.e., no one member has 
the capability to perform the total assessment — various members hold expertise in their disciplines and to provide 
the overall assessment, the members have to pool resources). Although, by being sequential, the individual sub- 
teams require minimal contact with each other. The team’s assessments are ultimately used in a hierarchical fashion 
(i.e., management uses the reports generated to make a decision/ recommendation), however, the team and its sub- 
teams interact in a non-hierarchical manner. While the individuals work relatively autonomously, they are 
sanctioned by the sequential design and the time frames imposed by the assessment process. 

ADTT 


ADTT is a web-based database developed at ARC. ISAT members using ADTT are able to “download” the data 
necessary for their particular discipline assessments and “upload” or publish the results of their assessment back to 



the central store. ADTT is used to manage the flow of the data and to assist in generating reports (as defined by 
specifications) for management and analyst use. 

Each step in the process will be discussed in turn below, but the essentials of ADTT are this. ADTT is designed to 
require the ISAT discipline analysts to wait for upstream assessments to be completed before downstream analysts 
may obtain the needed data. The data is stored and used in extended mark up language (.xml) an industry standard. 
This enables the data to be utilized by multiple applications. The individual analysts obtain the needed data and 
process it with the appropriate assessment tools within Model Center. 


Distribution 


Figure 1 illustrates the overall distribution of the ISAT team during the August workshop. The assessments took 
place across three NASA facilities (ARC, MSFC & LaRC). The team was operating in a semi-distributed fashion, 
meaning that individuals of each of the sub-teams were collocated, but the sub-teams (disciplines) were dispersed. 
This proved beneficial given the novelty of working outside of the MSFC CEC. 

The vehicle closure discipline was represented by 1 expert in CONSIZ working with a POST expert, 1 expert in 
INTROS also working closely with a POST expert (all located at LaRC) and 1 expert in HAVOC (this analyst was 
the only member at ARC). The safety & reliability discipline was represented by 1 expert also at LaRC. The 5 
discipline experts at LaRC were closely observed for the duration of the workshop. Activities of the remaining ISAT 
team members had to be inferred or inquired about. 

At MSFC, 2 disciplines were each comprised of several experts of distributed expertise. Operations involved 2 
experts working closely at the same workstation (it is difficult to infer the need for these two people) via NROC. 
Finally, the costs & economics discipline involved the use of at least 4 workstations ( 1 executive, and 3 sub- 
divisions — NAFCOM, IEM-optimistic, IEM-pessimistic). 


Pre-CEC activities 


Much work is done in preparation of the collaborative engineering prior to team meetings. In particular, individual 
technologists must prepare the data for inputting in their respective models. Several activities have occurred prior to 
the collaborative engineering assessment phase. 1) the reference vehicle has been defined and validated, 2) the 
technology data have been collected, and 3) the technology data have been used to modify the reference vehicle 
models. It is not clear how much “teamwork” is required of the members during the pre-CEC activities. 

Individual analyses 


POST-CONSIZ 


The POSTCONSIZ use case consists of 2 analysts working closely together (one POST analyst and one CONSIZ 
analyst). POST (Program to Optimize Simulated Trajectories) is program utilized for trajectory analysis. CONSIZ 
(CONfiguration Sizing) is a tool, which calculates weights and sizing data for analysis, conceptual design and 
preliminary design of launch vehicles. The analysts work individually on each tool, however the two need to come 
to agreement on the integration of the individual analyses. Thus, the two are always performed in tandem. 

These analyses (must be mentioned together) can be performed concurrently with POST-INTROS & HAVOC, but 
must be completed before downstream analyses can occur. See figure 2. Therefore, the POST-CONSIZ experts have 
a reciprocal relationship with each other, a pooled relationship with other weights and sizing analysts and form the 
initial step in a sequential chain of other disciplines (i.e., safety, operations, costs & economics, etc.) for that 
particular model. Currently, the POST analyst and the CONSIZ analyst had to work “over each other’s shoulder” on 
the same machine. 

Although the analyses are cognitive tasks by their nature, the sequences of events detailing the POST CONSIZ 
tasks would predominately fall within the execute quadrant of McGrath’s circumplex (McGrath, 1984). More 



specifically, they are executing performance tasks. Although, to some extent, a decision-making tasks occurs by the 
analysts in determining whether the data has converged. See Appendix A and Figure 3 for a description of the 
individual sequence of events performed by the sub-team. The checklist in the appendix represents the steps taken 
beginning with obtaining the data through publishing the data following analyses. The figure represents the analyses 
conducted while in possession of the data. 

POST-INTROS 


The POST-INTROS use case consists of 2 analysts working closely together (one POST analyst and one INTROS 
analyst). POST (Program to Optimize Simulated Trajectories) is program utilized for trajectory analysis. INTROS 
(INTegrated ROcket Sizing) is a tool which calculates weights and sizing data for analysis, conceptual design and 
preliminary design of launch vehicles. The analysts work individually on each tool, however the two need to come 
to agreement on the integration of the individual analyses. Thus, the two are always performed in tandem. 

These analyses (must be mentioned together) can be performed concurrently with POST-CONSIZ & HAVOC, but 
must be completed before downstream analyses can occur. See figure 2. Therefore, the POST-INTROS experts have 
a reciprocal relationship with each other, a pooled relationship with other weights and sizing analysts and form the 
initial step in a sequential chain of other disciplines (i.e., safety, operations, costs & economics, etc.) for that 
particular model. Currently, the POST analyst and the INTROS analyst had to work “over each other’s shoulder” on 
the same machine. 

Although the analyses are cognitive tasks by their nature, the sequences of events detailing the POST CONSIZ 
tasks would predominately fall within the execute quadrant of McGrath’s circumplex (McGrath, 1984). More 
specifically, they are executing performance tasks. Although, to some extent, a decision-making task occurs by the 
analysts in determining whether the data has converged — this is represented by McGrath’s choose quadrant. See 
Appendix B and Figure 4 for a description of the individual sequence of events performed by the sub-team. The 
checklist in the appendix represents the steps taken beginning with obtaining the data through publishing the data 
following analyses. The figure represents the analyses conducted while in possession of the data. 


HAVOC 


HAVOC (do not have the origin of this acronym) is a tool that can be performed by a single analyst to assess the 
trajectory and the weights and sizing similar to the above-mentioned models. This analysis was performed at ARC 
and was not physically observed. 

Similar to INTROS and CONSIZ, this model begins with the HAVOC analysis and is sequential for downstream 
disciplines (i.e., safety, operations, costs & economics, etc.). It may also be run concurrently with INTROS and 
CONSIZ. 

The checklist for this analyst (note only one individual in this task/cell) is found in appendix C. Since direct 
observations were not made, it must be inferred that the tasks of the HAVOC expert are similar to those of the other 
vehicle closure experts (i.e., trajectory and weights & sizing). Therefore, the sequences of events detailing the 
HAVOC tasks would predominately fall within the execute quadrant of McGrath’s circumplex (McGrath, 1984). 
More specifically, they are executing performance tasks. Again, it must be inferred that a decision-making task of 
assessing convergence occurs. 


Safety-reliability 

The safety and reliability discipline is represented by analysts from SAIC (Science Applications International 
Corporation), an organization contracted to develop a safety assessment tool. During the workshop, only one user 
occupied this position. 

Structurally, the safety analyst depends on upstream information to perform his (all analysts at LaRC during the 
workshop were male) task. Likewise, those disciplines downstream of the safety analyst depend on the output of the 



safety assessment. The safety discipline must run the same analysis on all three models (i.e., CONSIZ, INTROS, 
and HAVOC). This is provided an a priori decision has not been made as to which model will be used. For one 
assessment during the workshop, only the INTROS model was used. When only one analyst is used, an inherent 
bottleneck occurs in the sequence. That is, safety cannot perform any analyses prior to the completion of the vehicle 
closure team, and then is expected to perform analyses on all models while others downstream are awaiting his 
output. This is illustrated in Figure 2. While many analysts work on the vehicle closure column (5 in all), only one 
performs the assessments in the safety column. 

The activities are expressed in the checklist for the safety analysis in appendix D. Similar to the vehicle closure 
team, the analyses are cognitive tasks by their nature, but the sequences of events detailing the safety and reliability 
tasks’ would predominately fall within the execute quadrant of McGrath’s circumplex (McGrath, 1984). More 
specifically, they are executing performance tasks. 

COMET/ NROC 


This discipline was not directly observed. It is inferred that the individual tasks are similar to those of the previously 
mentioned disciplines. The operations assessment (conducted via NROC) is represented by the operations column of 
Figure 2 and addresses ground and flight operations costs. 

Appendix E displays the stepwise tasks performed by the analyses). Structurally, the operations discipline is 
sequentially interdependent with the other disciplines (i.e., see operations column of Figure 2.). It is not known 
whether 1 or more operators performed the analyses for this discipline. It is assumed that the operations discipline is 
similar to the safety discipline with respect to interdependence with others. 


NAFCOM-R/IEM 


This discipline was not physically observed. It is perhaps the most complicated of the disciplines. Has one 
workstation deemed the executive and then 3 separate stations for each sub-discipline (i.e., NAFCOM-costs; IEM 
pessimistic — economics; and, IEM optimistic-economics). IEM - (ISTP [Integrated Space Transportation Plan] 
Economics Model). NAFCOM tool predicts DDT&E (design, development, test and evaluation) and TFU 
(Theoretical first unit) cost estimates including the cost of subsystem and component hardware, system integration, 
scientific instruments and/or engine systems. 

In the sequential chain, this discipline is last. It is dependent on “good” data from upstream sources. It is not clear 
how much interaction must occur between the sub-disciplines. This discipline is represented by the final column of 
Figure 2. The figure shows that the discipline is further sub-divided into the 3 sub-teams. 


Subsequent actions 


Reports 

Management of the IS AT team generates reports and assesses the overall activities. Reports are also used by the 
members to (1) monitor their own activities — that is to quickly review their data, and (2) by members needing data 
from a discipline they are not expert. These reports can serve as team processes such as mutual performance 

monitoring. 

Recommendations 


Decisions are not made as to the feasibility of concept vehicles, but rather, recommendations are sent to 2 nd Gen. 
Decisions are made as to which model best assesses the vehicle (i.e., HAVOC, POST-CONSIZ, POST-INTROS). 
These decisions may be made a priori or at some point during or after the assessments. 



Sources 


Data comes from - observations, checklists, internal documents such as .ppts, .docs etc. 
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Figure 2. Model of workflow as dictated by ADTT parameters. (Each double-bordered block represents a separate analysis) 





























Figure 4 


Figure 4. Diagram of INTROS process. 
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POST-CONSIZ checklist for using ADTT 

1. Log onto the website. 

a. Enter the ADTT website at http://www.nas.nasa.gov/adttWeb . 

b. Login to your account by typing your login id and password and clicking on SUBMIT. 

c. Select the ISATProject and click on GO. 

2. Download the ModelCenter model file from the ADTT database onto your machine. 

a. Expand the Object Navigation Tree on the left of the page until the desired concept appears. This 
will be Bimese, Shuttle or Adv. Bimese 

b. Click on the desired Concept in the Object Navigation tree. 

c. Click on the Studies tab in the Content Area on the right. 

d. Click on the VIEW MATRIX link. 

e. Select the desired Mission and click DISPLAY. 

f. Check to see that all upstream analyses (i.e. all cells to the left of the Post-Consiz~VehicleClosure 
column) are either Not Required or Complete. If an upstream analysis is Required or In 
Progress, you cannot proceed any further. 

g. Click on the white cell marked REQUIRED in the Post-Consiz -VehicleClosure column, in the 
desired Vehicle row. Click this cell until it turns yellow. 

h. Click on the Set Cell to In-Progress button. 

i. The Cell will appear yellow and a link to the ModelCenter file containing Post-Consiz will be 
present. Right-click on this link and select ‘Save Target As...’ 

j. Save the ModelCenter file onto your local machine. Be sure to select the ‘All Files’ file type and 
provide the .pxc file extension. 

3. Run your application locally. 

a. Open Model Center and load the ModelCenter (.pxc) file you just downloaded. 

b. If a template file for CONSIZ needs to be changed, enter the new template file name next to the 
templateFile variable and make sure to reinitialize the component by right-clicking the 
component and click the “Reload Templates” button. 

c. A trajectory file is generated after a POST run. A plotting tool can be launched within Model 
Center to view the trajectory. 

d. Once the analysis is complete, be sure to run the dataDictOUT component at the end of your 
POST-INTROS Model. Save your Model File. 

4. Update the ADTT database by uploading your completed ModelCenter file. 

a. If you have logged out of the ADTT system, log back in. 

b. In your browser, expand the Object Navigation Tree on the left of the page until the desired 
concept appears. This will be Bimese, Shuttle or Adv. Bimese 

c. Click on the desired Concept in the Object Navigation tree. 

d. Click on the Studies tab in the Content Area on the right. 

e. Click on the VIEW MATRIX link. 

f. Select the desired Mission and click DISPLAY. 

g. Click on the Check In link in the cell you wish to upload to. This should be the cell representing 
the Post-Consiz ~VehicleClosure column for the vehicle you are about to upload to. Clicking the 
Check In link will pop up a window in the upper left of your screen. 

h. Browse your computer for the name of the ModelCenter file to upload. Type in a short comment. 
Click on Check-in ModelCenter File. Wait until the pop-up window says the upload was 
successful; it may take a while. 

i. Close the pop-up window when done. 

5. Refresh the Run Matrix. 

a. Click the ‘Refresh Matrix’ Button 

b. A link to the raw Post-Consiz data files will appear in the Completed cell 
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POST-INTROS checklist for using ADTT 

1. Log onto the website. 

a. Enter the ADTT website at htt p://www.nas.nasa.gov/adttWeb . 

t>. Login to your account by typing your login id and password and clicking on SUBMIT. 

c. Select the ISATProject and click on GO. 

2. Download the ModelCenter model file from the ADTT database onto your machine. 

a. Expand the Object Navigation Tree on the left of the page until the desired concept appears. This 
will be Bimese, Shuttle or Adv. Bimese 

b. Click on the desired Concept in the Object Navigation tree. 

c. Click on the Studies tab in the Content Area on the right. 

d. Click on the VIEW MATRIX link. 

e. Select the desired Mission and click DISPLAY. 

f. Check to see that all upstream analyses (i.e. all cells to the left of the Post-Intros- VehicleClosure 
column) are either Not Required or Complete. If an upstream analysis is Required or In 
Progress, you cannot proceed any further. 

g. Click on the white cell marked REQUIRED in the Post-Intros-VehicleClosure column, in the 
desired Vehicle row. Click this cell until it turns yellow. 

h. Click on the Set Cell to In-Progress button. 

i. The Cell will appear yellow and a link to the ModelCenter file containing Post-Intros will be 
present. Right-click on this link and select ‘Save Target As...’ 

j. Save the ModelCenter file onto your local machine. Be sure to select the ‘All Files’ file type and 
provide the .pxc file extension. 

3. Run your application locally. 

a. Open ModelCenter and load the ModelCenter (.pxc) file you just downloaded. 

b. To run INTROS in interactive mode, make sure that the userlnTheLoop flag for that component 
is set to true. 

c. If a template file for INTROS needs to be changed, enter the new template file name next to the 
templateFile variable and make sure to reinitialize the component by right-clicking the 
component and click the “reinitialize” button. 

d. When running INTROS in interactive mode, DO NOT close the EXCEL window. Always click 
the “OK” button in the pop up dialog box when finished. 

e. Try to modify input values in Model Center. If wrapped variables are changed after running 
interactive mode, make sure to right-click the component and click the “Reload Input Values” 
button. DO NOT modify any input variables that are linked from upstream components. Linked 
variables can not be reloaded. 

f. A trajectory file is generated after a POST run. A plotting tool can be launched within Model 
Center to view the trajectory. 

g. Once the analysis is complete, be sure to run the dataDictOUT component at the end of your 
POST-INTROS Model. Save your Model File. 

4. Update the ADTT database by uploading your completed ModelCenter file. 

a. If you have logged out of the ADTT system, log back in. 

b. In your browser, expand the Object Navigation Tree on the left of the page until the desired 
concept appears. This will be Bimese, Shuttle or Adv. Bimese 

c. Click on the desired Concept in the Object Navigation tree. 

d. Click on the Studies tab in the Content Area on the right. 

e. Click on the VIEW MATRIX link. 

f. Select the desired Mission and click DISPLAY. 

g. Click on the Check In link in the cell you wish to upload to. This should be the cell representing 
the Post-Intros-VehicleClosure column for the vehicle you are about to upload to. Clicking the 
Check In link will pop up a window in the upper left of your screen. 



h. Browse your computer for the name of the ModelCenter file to upload. Type in a short comment. 
Click on Check-in ModelCenter File. Wait until the pop-up window says the upload was 
successful; it may take a while. 

i. Close the pop-up window when done. 

5. Refresh the Run Matrix. 

a. Click the ‘Refresh Matrix’ Button 

b. A link to the raw Post-Intros data files will appear in the Completed cell 
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HAVOC checklist for using ADTT 

1. Log onto the website. 

a. Enter the ADTT website at http://www.nas.nasa.gov/adttWeb . 

b. Login to your account by typing your login id and password and clicking on SUBMIT. 

c. Select the ISATProject and click on GO. 

2. Download the ModelCenter model file from the ADTT database onto your machine. 

a. Expand the Object Navigation Tree on the left of the page until the desired concept appears. This will 
be Bimese, Shuttle or Adv. Bimese 

b. Click on the desired Concept in the Object Navigation tree. 

c. Click on the Studies tab in the Content Area on the right. 

d. Click on the VIEW MATRIX link. 

e. Select the desired Mission and click DISPLAY. 

f. Check to see that all upstream analyses (i.e. all cells to the left of the HAVOC-VehicleClosure 
column) are either Not Required or Complete. If an upstream analysis is Required or In Progress, you 
cannot proceed any further. 

g. Click on the white cell marked REQUIRED in the HAVOC~VehicleClosure column, in the desired 
Vehicle row. Click this cell until it turns yellow. 

h. Click on the Set Cell to In-Progress button. 

i. The Cell will appear yellow and a link to the ModelCenter file containing HAVOC will be present. 
Right-click on this link and select ‘Save Target As../ 

j. Save the ModelCenter file onto your local machine. Be sure to select the ‘All Files’ file type and 
provide the .pxc file extension. 

3. Run your application locally. 

a. Open ModelCenter and load the ModelCenter (.pxc) file you just downloaded. 

b. If necessary, load the desired xyz files into the uploadXYZ component in the Model. 

c. If necessary, load the desired HAVOC input file(s) into the appropriate HAVOC components. 

d. Run Havoc by clicking the green run icon in the upper left of HAVOC component. 

e. Once the analysis is complete, be sure to run the dataDictOUT component at the end of your Havoc 
Model. 

f. Save your Model File. 

4 . Update the ADTT database by uploading your completed ModelCenter file. 

a. If you have logged out of the ADTT system, log back in. 

b. In your browser, expand the Object Navigation Tree on the left of the page until the desired concept 
appears. This will be Bimese, Shuttle or Adv. Bimese 

c. Click on the desired Concept in the Object Navigation tree. 

d. Click on the Studies tab in the Content Area on the right. 

e. Click on the VIEW MATRIX link. 

f. Select the desired Mission and click DISPLAY. 

g. Click on the Check In link in the cell you wish to upload to. This should be the cell representing the 
HAVOC~VehicleClosure column for the vehicle you are about to upload to. Clicking the Check In 
link will pop up a window in the upper left of your screen. 

h. Browse your computer for the name of the ModelCenter file to upload. Type in a short comment. 
Click on Check-in ModelCenter File. Wait until the pop-up window says the upload was successful; it 
may take a while. 

i. Close the pop-up window when done. 

5. Refresh the Run Matrix. 

a. Click the ‘Refresh Matrix’ Button 

b. A link to the raw HAVOC data files will appear in the Completed cell 
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SAIC safety checklist for using ADTT 

1. Log onto the website. 

a. Enter the ADTT website at http://www.nas.nasa.gov/adttWe_b . 

b. Login to your account by typing your login id and password and clicking on SUBMIT. 

c. Select the ISATProject and click on GO. 

2. Download the ModelCenter model file from the ADTT database onto your machine. 

a. Expand the Object Navigation Tree on the left of the page until the desired concept appears. This will 
be Bimese, Shuttle or Adv. Bimese 

b. Click on the desired Concept in the Object Navigation tree. 

c. Click on the Studies tab in the Content Area on the right. 

d. Click on the VIEW MATRIX link. 

e. Select the desired Mission and click DISPLAY. 

f. Check to see that all upstream analyses (i.e. all cells to the left of the SAIC~Safety column) are either 
Not Required or Complete. If an upstream analysis is Required or In Progress, you cannot proceed 
any further. 

g. Click on the white cell marked REQUIRED in the SAIC~Safety column, in the desired Vehicle row. 
Click this cell until it turns yellow. 

h. Click on the Set Cell to In-Progress button. 

i. The Cell will appear yellow and a link to the ModelCenter file containing the SAIC safety tool will be 
present. Right-click on this link and select ‘Save Target As...’ 

j. Save the ModelCenter file onto your local machine. Be sure to select the ‘All Files’ file type and 
provide the .pxc file extension. 

k. You do not need to download the DataDictionary as the Ops group has done that for you. 

3. Run your application locally. 

a. Open ModelCenter and load the .pxc file you just downloaded. 

b. Select and set the appropriate inputs for the component that you are going to run from those exposed 
within the component in ModelCenter. 

i. ModelCenter will automatically load values for the following variables: 

1 . Launch Vehicle Type 

2. Case Name 

3 . Number of Booster Engines 

4. Number of Orbiter Engines 

5. Booster Fuselage Area 

6. Orbiter Fuselage Area 

7. Booster Wing Area 

8. Orbiter Wing Area 

9. Booster Body Flap Area 

10. Orbiter Body Flap Area 

1 1 . Booster Control Surface 

12. Orbiter Control Surface 

13. Booster Power Level 

14. Orbiter Power Level 

ii. The Safety Analyst running the component should review the following variables and update 
them to match the current case (drop down boxes provide a list of choices where appropriate): 

1 . Booster Engine T ype 

2. Orbiter Engine Type 

3. Crew Transfer Vehicle 

4. Auxiliary Power Type 

5. OMS Propellant Type 

6. RCS Propellant Type 

7. Advanced Structures 

8. Advanced Undercarriage 

9. Blanket TPS 

10. CMC Control Surfaces 



1 1 . Composite Tanks 

12. Densified Propellants 

13. Electromechanical Actuators 

14. Power Management and Distribution 

15. IVHM 

16. PEM Fuel Cells 

17. Rapid Turnaround TPS 

c. Run the desired component. Average runtime is 90 seconds. 

d. Once all analysis is complete, you need to run the dataDictOut component. This is done by first using 
the right mouse click on the dataDictOut component icon within the main component view window, 
and selecting the Reload Templates from the menu. Next hit yes to update the variables and then run 
the component. 

4 . Update the ADTT database by uploading your completed ModelCenter file. 

a. If you have logged out of the ADTT system, log back in. 

b. In your browser, expand the Object Navigation Tree on the left of the page until the desired concept 
appears. This will be Bimese, Shuttle or Adv. Bimese 

c. Click on the desired Concept in the Object Navigation tree. 

d. Click on the Studies tab in the Content Area on the right. 

e. Click on the VIEW MATRIX link. 

f. Select the desired Mission and click DISPLAY. 

g. Click on the Check In link in the cell you wish to upload to. This should be the cell representing the 
SAIC~Safety column for the vehicle you are about to upload to. Clicking the Check In link will pop up 
a window in the upper left of your screen. 

h. Browse your computer for the name of the ModelCenter file to upload. Type in a short comment. 

Click on Check-in ModelCenter File. Wait until the pop-up window says the upload was successful; it 
may take a while. 

i. Close the pop-up window when done. 

5 . Refresh the Run Matrix. 

a. Click the ‘Refresh Matrix’ Button 

b. A link to the raw SAIC~Safety data files will appear in the Completed cell 
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NROC checklist for using ADTT 

1 . Log onto the website. 

a. Enter the ADTT website at http://www.nas.nasa.gov/adttWeb , 

b. Login to your account by typing your login id and password and clicking on SUBMIT. 

c. Select the ISATProject and click on GO. 

2. Download the ModelCenter model file from the ADTT database onto your machine. 

a. Expand the Object Navigation Tree until the desired concept appears. This will be Bimese, Shuttle or 
Adv. Bimese 

b. Click on the desired Concept in the Object Navigation tree. 

c. Click on the Studies tab in the Content Area on the right. 

d. Click on the VIEW MATRIX link. 

e. Select the desired Mission and click DISPLAY. 

f. Check to see that all upstream analyses (i.e. all cells to the left of the COMET-Operations column) are 
either Not Required or Complete. If an upstream analysis is Required or In Progress, you cannot 
proceed any further. 

g. Click on the white cell marked REQUIRED in the COMET-Operations column, in the desired 
Vehicle row. Click this cell until it turns yellow. 

h. Click on the User Save Checked-out Cell button. 

i. Refresh the table by repeating steps 1-c and 1-d above. The In Progress cell should now contain a 
link to the appropriate ModelCenter and XML data dictionary files. 

j. Download the ModelCenter and XML data dictionary files to your local machine by right-clicking on 
the links and saving the files to the locations of your choice on your local machine. 

3. Run your application locally. 

a. Open ModelCenter and load the .pxc file you just downloaded. 

b. Load the XML data dictionary into the model you just opened by first exposing the first level of 
variables under the dataDictln component within the component list on the left hand side of your 
ModelCenter view. Right click the rawlnput variable and select “load from file” from the menu. 
Browse and select the XML data dictionary file that you just downloaded. 

c. Run the dataDictln component. 

d. Select and set the appropriate NROC inputs from those exposed within the component in ModelCenter. 

e. Run the NROC component. If the component is run interactively set the appropriate input parameter 
(for example in the NoFeeTo sheet) to NROC. 

f. If the NROC sheet was run in interactive mode finish the NROC run by hitting the OK button on the 
vbScript dialog (DO NOT EXIT DIRECTLY FROM EXCEL). 

g. Save the ModelCenter model file to the same or new file name. 

4. Update the ADTT database by uploading your completed ModelCenter file. 

a. If you have logged out of the ADTT system, log back in. 

b. In your browser, expand the Object Navigation Tree on the left of the page until the desired concept 
appears. This will be Bimese, Shuttle or Adv. Bimese 

c. Click on the desired Concept in the Object Navigation tree. 

d. Click on the Studies tab in the Content Area on the right. 

e. Click on the VIEW MATRIX link. 

f. Select the desired Mission and click DISPLAY. 

g. Click on the Check In link in the cell you wish to upload to. This should be the cell representing the 
COMET-Operations column for the vehicle you are about to upload to. Clicking the Check In link 
will pop up a window in the upper left of your screen. 

h. Browse your computer for the name of the ModelCenter file to upload. Type in a short comment. 

Click on Check-in ModelCenter File. Wait until the pop-up window says the upload was successful; 
it may take a while. 

i. Close the pop-up window when done. 

5. Refresh the Run Matrix. 

a. Click the ‘Refresh Matrix’ Button 

b. A link to the raw COMET data files will appear in the Completed cell 
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NAFCOM-R/IEM checklist for using ADTT 

1. Log onto the website. 

a. Enter the ADTT website at http://www.nas.nasa.gov/adttWeb , 

b. Login to your account by typing your login id and password and clicking on SUBMIT. 

c. Select the ISATProject and click on GO. 

2. Download the ModelCenter model file from the ADTT database onto your machine. 

a. Expand the Object Navigation Tree on the left of the page until the desired concept appears. This 
will be Bimese, Shuttle or Adv. Bimese 

b. Click on the desired Concept in the Object Navigation tree. 

c. Click on the Studies tab in the Content Area on the right. 

d. Click on the VIEW MATRIX link. 

e. Select the desired Mission and click DISPLAY. 

f. Check to see that all upstream analyses (i.e. all cells to the left of the NAFCOM-IEM~Cost-Econ 
column) are either Not Required or Complete. If an upstream analysis is Required or In 
Progress, you cannot proceed any further. 

g. Click on the white cell marked REQUIRED in the NAFCOM-IEM~Cost-Econ column, in the 
desired Vehicle row. Click this cell until it turns yellow. 

h. Click on the Set Cell to In-Progress button. 

i. The Cell will appear yellow and a link to the ModelCenter file containing NAFCOM-IEM will be 
present. Right-click on this link and select ‘Save Target As...’ 

j. Save the ModelCenter file onto your local machine. Be sure to select the ‘All Files’ file type and 
provide the .pxc file extension. 

k. You do not need to download the DataDictionary as the Ops group has done that for you. 

3. Run your application locally. 

a. Open ModelCenter and load the .pxc file you just downloaded. 

b. Select and set the appropriate inputs for the component that you are going to run from those 
exposed within the component in ModelCenter. 

c. Run the desired component. In some cases you may have to wait for an upstream tool within the 
model to be run first. 

d. If the component is run interactively set the appropriate input parameters. 

e. If the component was run in interactive mode, finish the excel instance by hitting the OK button 
on the vbScript dialog (DO NOT EXIT DIRECTLY FROM EXCEL). 

f. Once all analysis is complete, you need to run the dataDictOut component. This is done by first 
using the right mouse click on the dataDictOut component icon within the main component view 
window, and selecting the Reload Templates from the menu. Next hit yes to update the variables 
and then run the component. 

4. Update the ADTT database by uploading your completed ModelCenter file. 

a. If you have logged out of the ADTT system, log back in. 

b. In your browser, expand the Object Navigation Tree on the left of the page until the desired 
concept appears. This will be Bimese, Shuttle or Adv. Bimese 

c. Click on the desired Concept in the Object Navigation tree. 

d. Click on the Studies tab in the Content Area on the right. 

e. Click on the VIEW MATRIX link. 

f. Select the desired Mission and click DISPLAY. 

g. Click on the Check In link in the cell you wish to upload to. This should be the cell representing 
the NAFCOM-IEM~Cost-Econ column for the vehicle you are about to upload to. Clicking the 
Check In link will pop up a window in the upper left of your screen. 

h. Browse your computer for the name of the ModelCenter file to upload. Type in a short comment. 
Click on Check-in ModelCenter File. Wait until the pop-up window says the upload was 
successful; it may take a while. 

i. Close the pop-up window when done. 

5. Refresh the Run Matrix. 

a. Click the ‘Refresh Matrix’ Button 

b. A link to the raw NAFCOM-IEM data files will appear in the Completed cell 
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Executive Summary 

Guided by a literature review on team processes from several domains (e.g., 
organizational and human factors psychology, and engineering management), a general 
model of team processes was developed. An engineering task analysis of a specific 
engineering team (Inter-center Systems Analysis Team of the National Aeronautics & 
Space Administration - ISAT) was used to develop an engineering specific model of 
team processes. The model (see figure 5, pg. 11) includes the following processes relevant 
to engineering teams. 

■ Communication ■ Mutual Performance Monitoring 

■ Coordination ■ Intra-team Feedback 

Measures were given to the ISAT members following a November workshop. The ISAT 
members scored themselves rather highly on two of the processes (i.e., communication & 
coordination) as well as in performance. Table 1 (page 15) shows those results. The 
survey demonstrated that the members do not agree on the extent of monitoring and 
feedback that occurs within the team. This cross-sectional observation indicates that the 
team processes of monitoring and feedback may be adversely affected by geographic 
distribution. Further enhancements of collaborative technology for distributed 
engineering should consider the impact of monitoring and feedback on team performance 
and make appropriate accommodations (e.g., automate monitoring of data, automate 
report generation). 
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Introduction 

In a comprehensive review of the teamwork literatures, including organizational 
psychology, human factors psychology and engineering management, Fletcher and Major 
(2001) reported the existence of 12 conceptually distinct processes relevant to team 
effectiveness in general - each empirically supported in the literature. They also noted a 
host of moderators (e.g., task, team and individual characteristics) influencing the 
salience of each of the processes for different teams. That is, not all 12 processes are 
relevant for all teams at all times given team composition and characteristics. Recent 
advances in technology have allowed teams to work while the members are 
geographically dispersed, yet little is known about the effects of such dispersion on these 
team processes. Using a task analysis of a particular engineering team (i.e., Inter-center 
Systems Analysis Team of the National Aeronautics & Space Administration - ISAT) 
during workshops conducted in August 2001, an engineering process model was 
developed. Measures of team process were taken in a subsequent workshop of ISAT in 
November 2001. This report focuses on the development and validation of the 
engineering team process model in distributed collaborative environments. 

Engineering Process Model 

The general team process model depicted in Figure 1 is used to illustrate the role of 
interdependence and team processes on a team’s performance. The model is generalizable 
to teams found in a variety of settings (e.g., sports teams, engineering teams, cockpit 
crews, etc.). Another model implicitly guiding the current discussion is depicted in 
Figure 2. The team process improvement model demonstrates a cycle of team assessment. 
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Figure 1. The general team process model. 


A team must understand who is considered part of the team, what they do (and who relies 
on whom) and how they should best collaborate (i.e., teamwork behaviors that facilitate 
the necessary interactions). Finally, the model depicts a continuous cycle of 
implementation of the processes (e.g., through measurement and training). Through 
assessment of team behaviors/processes, one can determine if training is needed or if 
infrastructure changes would better accommodate the team. When any changes occur 
within the team or in the team’s context (e.g., organizational constraints, technological 
implementation), then the assessment cycle should begin anew. That is, if the team’s 
composition changes, or if the team task is altered, then another assessment should be 
undertaken. 

A similar approach was taken in an assessment of ISAT. A task analysis of ISAT in a 
semi-distributed environment (Fletcher, 2001b) revealed that the individual sub-units 
(i.e., assessment disciplines) were largely sequentially interdependent. Figure 3 depicts 
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Figure 2. Team Process Improvement Model. 

the general flow of work for each analysis and therefore the nature of the 
interdependence. Following the implementation of Product Data Management (PDM), 
several of the disciplines became parallel in nature. That is, the overall task remained 
sequential, starting with vehicle closure and ending with an economics assessment, but 
the newly structured team allows three disciplines to work simultaneously and 
independently for a particular assessment. Figure 4 depicts the new arrangement. This 
new work structure alleviates some of the interdependence among the disciplines (e.g., 
NAFCOM no longer ‘waits’ for FIRST to be completed before beginning its phase of the 
assessment). This being said, team processes that were salient before are expected to 
remain important to team performance and effectiveness. 
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General Team Process Model 

The general team process model depicted in Figure 1 is an input - throughput - 
output model. Inputs include all contributions of the individual units that compose the 
team. These individual units may be smaller sub-units (i.e., side-bars, smaller teams, etc.) 
or individual team members. Each possesses a set of dispositions, knowledge, skills and 
abilities (KSAs), and level of motivation to perform the team task (Fletcher & Major, 
2001). The output of the model represents the team’s accomplishments (i.e., performance 
effectiveness both in terms of quality and quantity of output). This output is determined 
by the inputs, but partially mediated by the throughput variables. In the model, if 
individuals can contribute to the team outcome without the interaction of others (i.e., little 
interdependence) then team processes are less relevant to the team’s performance. 
However, if interdependence exists, then team processes are important in influencing 
team effectiveness. By determining where the interdependencies among team members 
exist and applying the team processes that can best facilitate the team, performance can 
be enhanced. 

ISAT Model of Team Processes 

A first step in determining the influence of team processes and how they might be 
inhibited is to assess the team’s interdependencies (i.e., where members rely on and 
interact with other members). The arrows in Figure 3 (i.e., ISAT in August — prior to 
PDM implementation) not only represent the flow of work for the ISAT team, they also 
indicate interactions among the disciplines (i.e., subunits). The boxes in Figure 3 (i.e., 
POST-CONSIZ, FIRST, NROC, etc.) represent the tools associated with a particular sub- 
unit. The members may not directly interact with another member (e.g., the FIRST user 




Figure 3. Collaborative engineering and task interdependence: the IS AT case (August, 2001). 
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Figure 4. Revised ISAT interdependence model - parallel disciplines (November, 2001). 


may not communicate directly with the HAVOC user), but they are dependent on each 
other. Specifically in a sequential fashion - those downstream must wait for and depend 
on the accuracy of information to be processed by those upstream. The revised 
interdependencies model, Figure 4, shows a more parallel distribution, but the 
interdependencies remain largely sequential (e.g., the FIRST user must wait on the 
POST-CONSIZ user, the IEM user must wait and depends on accuracy from all upstream 
analysts). The interdependencies are largely based on the collection and dissemination of 
data from the data dictionary. 

Figure 5 lists the main tasks associated with the interchange of data. The tasks are 
specific behaviors representing interactions among members (i.e., in this case, 
interactions associated with collection and dissemination of data). A matrix was 
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developed with these tasks along the rows; columns represent each of four specific team 
processes observed to be related to performance outcomes (i.e., observed from task 
analysis and critical incidents of ISAT performance). The processes associated with ISAT 
during actual collaborative tasks are communication, coordination, monitoring and 
feedback. An V is placed in each cell where a process is most relevant to improving or 
detracting from performance. For example, communication and coordination are 
important when obtaining the proper file to be assessed (i.e., so that the proper vehicle is 
assessed). Monitoring and feedback are likewise important when ‘uploading’ the data 
once it has been assessed to ensure quality control. For instance, errors made upstream 
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can prove costly for downstream members (e.g., when errors are caught all affected may 
have to re-perform their analyses). Instituting these four processes - or ensuring their 
viability - would improve efficiency, reduce haphazard errors, and reduce wait time. This 
approach to model development could be generalized to most any team. Likewise, Figure 
5 only represents a portion of the IS AT team’s individual disciplines. It is logical to 
assume that the model may be extended to all discipline interactions to the extent that 
such interactions are similar. 

Model Assessment 

There are a variety of methods available to measure human behavior (e.g., 
observation, self-report, peer-report, etc.). Each has costs and benefits. While it may be 
optimal to have an objective observational method in place in which expertly trained 
observers rate individuals in certain areas, it is not always practical. Therefore, those 
interested in assessing psychological constructs often use self-report data. There exists a 
general problem with self-report data, however, in that in some instances, individuals 
tend to rate themselves spuriously high in comparison to peer or supervisors’ ratings (see 
Mount & Scullen, 2001). This may be due to a better understanding of oneself (as may be 
the case with measures of ‘hidden’ constructs such as motivation), due to the desire of the 
individual to be viewed in a more favorable light, or potentially, due to geographic 
dispersion (e.g., when assessing non-col located team members - team constructs). We 
chose to utilize self-report assessments to demonstrate proof of principle of the 
engineering process model and the effects of geographic dispersion on collaborative 
engineering. For that purpose, self-assessments proved useful. 
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Team Process Measures 

The ISAT-specific engineering process model (Figure 5) identifies four team 
processes relevant to team performance. Those include communication, coordination, 
monitoring and feedback. Rosenstein (1994) reported on the psychometric properties of 
scales measuring these constructs. The measures (see Appendix) were adopted for use 
with ISAT. Rosenstein (1994) demonstrated construct validity of the measures (i.e., there 
is evidence that the tests measure their purported constructs) and provided congeneric 
reliabilities for the scales: .91 for communication, .81 for coordination, .73 for 
monitoring and .81 for feedback. Each of these exceeds the recommendation by Nunnally 
and Bernstein (1994) of using scales with a minimum internal consistency of .70. The 
measure provides a definition of the construct (e.g., communication) and asks each team 
member to rate a set of items on a scale of 1 (almost never) to 5 (almost always) 
regarding how often team members engaged in each behavior. 

Another issue that arises in using self-report data for team level constructs is the issue 
of agreement and aggregation. Rousseau and others (Rousseau, 1985; Roberts, Hulin & 
Rousseau, 1978) argued that it is appropriate to use individual level data to represent a 
higher-level (e.g., a team, workgroup, etc.) only when sufficient evidence exists as to the 
agreement of those individuals. A simple example will clarify this. Suppose that several 
members of a team were asked to rate the team’s level of cohesiveness on a scale of 1 
(not at all) to 5 (extremely). If half of the members rated the team a 5 and the other half a 
1, their average would be a 3. One would ask if “3” was an accurate depiction of the 
team’s cohesiveness. Certainly there is disagreement among the members, and the 
average is not meaningful in depicting the team level construct. However, the 
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disagreement may prove useful in identifying problem areas for the team (i.e., why is 
there disagreement - communication problems? in-group/out-group issues?). 

James, Demaree and Wolf (1984, 1993) developed a method for assessing within- 
group agreement (r W G)- The procedure that they proffered compares the variance 
associated with the individual raters and items with a theoretical null variance that would 
be expected due to chance. Within-group agreement is calculated by the following 
formula: 

where 7 2 is the mean of the observed variances on J essentially parallel items, and o E u 

■V 

is the variance on xj that would be expected if all ratings were due exclusively to random 
measurement error - the null variance. James et al. (1984) provided a method for 
determining the theoretical null variance for a variety of occasions (e.g., a rectangular, 
uniform distribution, triangular distribution in which central tendency bias is present, and 
a negatively skewed distribution in which there is a bias towards positive ratings). 

It is reasonable to assume that members of a team may rate themselves in a favorable 
light. That is, in assessing psychological constructs related to team performance, the 
distribution of self-report scores may be negatively skewed (i.e., a positive bias). 
Therefore, in assessing twg it is appropriate to evaluate the null distribution based on 
such a hypothesis. James et al. (1984) suggested using .90 for a null variance with a 
moderately negative skew when a 5-point rating scale is used. 

A rule-of-thumb has been suggested that values for twg of -70 or greater sufficiently 
justify aggregation of data to the higher-level (Cohen, Doveh, & Eick, 2001). This means, 
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given sufficient agreement of the team members (i.e., twg > -70) then the mean of their 
individual scores may be used to represent the team-level construct. 

Team Processes - ISAT 

The team process measures were distributed to the ISAT members during a 
November 2001 workshop. This workshop was conducted to familiarize the members 
with the deployment of PDM to further enable distributed collaboration. The general 
findings are presented in Table 1 . In all, five individual members responded to the survey 
representing four ISAT disciplines (i.e., vehicle closure, safety & reliability, costs, and 
operations). The overall purpose of the self-assessment was to gain an understanding of 
the effect of the geographic distribution of the members and the technology used on the 
relevant team processes. 


Table 1. Summary of ISAT responses to team process measures. 



Communication 

Mutual 

Performance 

Monitoring 

Intra-team 

Feedback 

Coordination 

Performance 

N 

.33 

.89 

.82 

.68 

.42 

twg 

.95 

.10 

.46 

.75 

.91 

Mean 

4.27 

** 

** 

3.87 

4.24 


Note, twg was calculated using a moderately skewed null distribution. Anchors for the 
scales were 1 (almost never) to 5 (almost always). 

** Due to lack of agreement, calculation of a mean is not appropriate for representing the 


team-level construct. 
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The ISAT members indicated that two of the team processes and a self-rating of 
performance were high. They were in relative agreement (i.e., twg > -75) for 
communication, coordination and performance and indicated a generally high score 
(mean > 3.9) for those three measures. As for mutual performance monitoring and intra- 
team feedback, the members were in disagreement (rwG - -10 and .46 respectively). This 
prohibits us from using the mean scores for these constructs in a meaningful manner. 
However, the fact that there is a discrepancy may be helpful in determining team needs. 
This may illustrate problem areas that can be improved with infrastructure changes (e.g., 
technological advancements facilitating mutual performance monitoring). 

Concluding Remarks 

The general aim of this project has been to develop a methodology for the assessment 
and continuous improvement of engineering team effectiveness in distributed 
collaborative environments. This has largely been accomplished through the observations 
and assessment of a specific engineering team, ISAT, as it transitioned from a fully 
collocated team to a geographically dispersed team. Relevant processes from several 
domains (i.e., organizational & human factors psychology) were used to develop a model 
that proved helpful in identifying team needs. 

The team processes, communication, mutual performance monitoring, intra-team 
feedback, and coordination were deemed relevant to ISAT’s performance from both a 
thorough review of the literature and appropriate team task analysis. These processes 
were used to develop an engineering team process model specifically for ISAT (see 
Figure 5). This model is composed of a matrix with the team tasks associated with 
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interdependence orthogonal to team processes. The model is important in that it identifies 
both areas of interdependence and the nature of the team processes relevant to those 
interdependencies. A similar model would be developed for other teams using the 
approach outlined in the Team Process Improvement Model (Figure 2). 

Once the processes salient to team effectiveness had been identified, we were able to 
assess the team in terms of how the processes were affected by the team’s recent changes 
(i.e., geographic dispersion). That is, team members were surveyed using a self-report 
instrument. Information from this questionnaire identified problem areas for the team. In 
this case, team process barriers were identified by the members relative disagreement in 
conjunction with critical incidents (Fletcher, 2001a). Team members were not in 
agreement on the extent of mutual performance monitoring and intra-team feedback in 
the team. This could likely be improved with advancements in the collaborative tools that 
they use. For instance, by implementing a monitoring system within the PDM (or other 
technology) data integrity may be automatically monitored and reports could be 
automated. Of course, these are only recommendations and are not the only plausible 
reasons for team member disagreement. It should not be overlooked that the members 
view the team as high in communication and coordination despite the challenge of 
geographic dispersion. 

Overall, the aim has been met. A methodology of continuous assessment aimed at 
improving team process has been proposed. Further, by using a particular engineering 
team, ISAT, we were able to demonstrate that a team specific model could be generated 
to detail the salience of certain team processes. By using a psychometrically sound self- 
report instrument, we were able ‘ask’ the particular team members how they were 
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affected by geographic dispersion - simultaneously demonstrating proof of principle for 
the methodology and identifying particular concerns for the collaborative engineering 
team in distributed environments. 
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Appendix 

Teamwork Components Rating Scale 

Instructions: 

Using the rating scale below, please rate the teamwork behaviors you saw the ISAT team 
exhibiting. Please think of the most recent assessment; include experts from all models and 
disciplines involved in the assessment when considering team members. This questionnaire is for 
developmental purposes only; your responses will remain confidential. Insert your answers into 
the document; save; and e-mail as an attachment to tom.fletcher@verizon.net . 


Almost Never 

1 

Sometimes 

1 

Almost Always 
1 1 

1 

2 

3 

4 5 


Write “N/A” if a behavior does not apply 

Communication: Communication involves the exchange of information between two or more 
team members in the prescribed manner and by using proper terminology. Often the purpose of 
communication is to clarify or acknowledge the receipt of information. 

Team members: 

1. Clarify intentions to other team members. 

2. Clarify procedures in advance of assignments. 

3. Pass complete information as prescribed. 

4. Acknowledge and repeat messages to ensure understanding. 

5. Communicate with proper terminology and procedures. 

6. Verify information prior to making a report. 

7. Ask for clarification of performance status when necessary. 

8. Follow proper communication procedures in passing and receiving information. 

9. Ensure that members who receive information understand it as it was intended to be 

understood. 

10. Communicate information related to the task. 

1 1 . Discuss task-related problems with others. 

Monitoring: Monitoring refers to observing the activities and performance of other team 
members. It implies that team members are individually competent and that they may 
subsequently provide feedback and backup behavior. 

Team members: 

1 . Are aware of other team members’ performance. 

2. Are concerned with the performance of the team members with whom they interact 

closely. 

3. Make sure other team members are performing appropriately. 

4. Recognize when a team member makes a mistake. 

5. Recognize when a team member performs correctly. 

6. Notice the behavior of others. 

7. Discover errors in the performance of another team member. 

8. Watch other team members to ensure that they are performing according to guidelines. 

9. Notice which members are performing their tasks especially well. 
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Feedback: Feedback involves the giving, seeking, and receiving of information among members. 
Giving feedback refers to providing information regarding other members’ performance. Seeking 
feedback refers to requesting input or guidance regarding performance. Receiving feedback 
refers to accepting positive and negative information regarding performance. 

Team Members: 

1. Respond to other members’ requests for performance information. 

2. Accept time-saving suggestions offered by other team members. 

3. Explain terminology to a member who does not understand its meaning. 

4. Ask the supervisor for input regarding their performance and what needs to be worked 

on. 

5. Are corrected on a few mistakes, and incorporate the suggestions into their procedures. 

6. Use information provided by other members to improve behavior. 

7. Ask for advice on proper procedures. 

8. Provide helpful suggestions to other members. 

9. Provide insightful comments when an assignment does not go as planned. 

Coordination: Coordination refers to team members executing their activities in a timely and 
integrated manner. It implies that the performance of some team members influences the 
performance of other team members. This may involve an exchange of information that 
subsequently influences another member’s performance. 

Team members: 

1 . Complete individual tasks without error, in a timely manner. 

2. Pass performance-relevant data from one to another in an efficient manner. 

3. Are familiar with the relevant parts of other members’ jobs. 

4. Facilitate the performance of each other. 

5. Carry out individual tasks in synchrony. 

6. Cause each other to work effectively. 

7. Avoid distractions during critical assignments. 

8. Carry out individual tasks effectively thereby leading to coordinated team 

performance. 

9. Work together with other members to accomplish team goals. 

Performance: Performance concerns the accomplishment of the activities and tasks required of 
the team. This team performance occurs with a consideration of the goals and expectations of 
team members, the supervisor, and the larger organization. 

Team members: 

1. Accomplish team goals. 

2. Meet or exceed expectations of the team. 

3. Meet performance goals in a timely manner. 

4. Regard team output as adequate or acceptable. 

5. Achieve team goals with few or no errors. 

6. Produce team output that meets standards of the organization. 

7. Regard accomplishments of the team to be above average. 

8. Feel that the team as a whole performed at an acceptable level. 

9. Met team objectives in an efficient manner. 
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Discipline: Vehicle Closure Safety & Reliability Operations Costs & 

Economics 

Please indicate tool used: 

Vehicle: TSTO parallel glideback TSTO parallel bum flyback TSTO serial 

flyback 
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Abstract. This paper describes a system of systems or metasystems approach and models 
developed to help prepare engineering organizations for distributed engineering environments. 
These changes in engineering enterprises include competition in increasingly global 
environments; new partnering opportunities caused by advances in information and 
communication technologies, and virtual collaboration issues associated with dispersed teams. 
To help address challenges and needs in this environment, a framework is proposed that can be 
customized and adapted for NASA to assist in improved engineering activities conducted in 
distributed, enhanced engineering environments. The approach is designed to prepare engineers 
for such distributed collaborative environments by learning and applying e-engineering methods 
and tools to a real-world engineering development scenario. The approach consists ot two 
phases: an e-engineering basics phase and e-engineering application phase. The e-engineering 
basics phase addresses skills required for e-engineering. The e-engineering application phase 
applies these skills in a distributed collaborative environment to system development projects. 

1. Introduction 

1.1. Background 

The effects of globalization are dramatically changing the practice of engineering and 
technology in the areas of enterprise project activities and advanced engineering environments. 
The National Research Council’s (NRC’s) Committee on Advanced Engineering Environments 
expects further significant changes in engineering product design, project processes, as well as 
collaboration support, education and training within the near future (NRC, 2000). Product 
design and analysis is increasingly using web-based systems to assist the communication 
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between distributed team members. Attempts are being made to collapse project processes in 
terms of steps and time requirements in order for enterprises to increase engineering team 
efficiency. Organizations, such as the National Aeronautics & Space Administration (NASA), 
are conducting pilot enhanced engineering initiatives to help assess whether design and analysis 
teams can be distributed and more engineering activities combined or conducted in parallel to 
compress the resources and time required for front-end engineering efforts. Distributed 
collaboration systems to support such efforts are growing more and more complex, including 
grid-like network infrastructures connecting team members with secure high bandwidth, 
shareable distributed engineering data artifacts, distributed engineering tool sharing, and 
synchronous audio and video. 

1.2. Global e-engineering environment 

This resulting global enterprise environment is complex, dynamic, and produces many 
collaboration challenges among product development and manufacturing teams, as shown in 
Figure 1. Global presence means geographically distributed team members from diverse 
organizational and national cultures. Global organizations and collapsed project engineering 
cycles can create team instability as various skills are quickly applied to product design 
challenges. Unfamiliarity between team members is more likely due to less face-to-face 
interaction. Project characteristics will include reduced development cycles, greater engineering 
complexity, increased integration, and tighter budgets. Generating success in the new reality of 
global enterprises is much different than what was required in traditional engineering 
environments. Enterprises will need to transform and ensure product development teams thrive 
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in a virtual collaborative engineering environment. This environment and the teams working in 
it must be capable of high collaborative performance conducive to innovation within dynamic 
schedule, cost, and performance constraints. 


Geographic Diverse Member Team 

distribution cultures unfamiliarity instability 

uu 


Global Product Development Environment 



( ( ( ( 

Short f combined Engineering Increased Tighter 

development complexity integration budgets 
cycles 

Figure 1 . Global product development environment challenges. 


We are calling teamwork in this environment, e-engineering, which is defined as ‘distributed 
collaboration in cyberspace using leading edge technologies enabling physically-dispersed, 
diverse teams to create integrated, innovative and competitive products, systems, and services.’ 
According to National Research Council studies (1999, 2000), an ideal virtual collaborative 
engineering environment or Advanced Engineering Environment (AEE) would accommodate 
diverse user groups and facilitate their collaboration by helping to eliminate cultural barriers 
between groups from different parts of an organization, different organizations, or different areas 
of the world. There are a number of important benefits that can be achieved from effective team 
use of virtual collaborative engineering technologies and methods as follows (Mills, 1998): 
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• Lower product development, design and production costs. Cost is the first and foremost 
factor driving much of the interest in VCE technologies. Products can be developed with 
more interaction in less time at a reduced cost. This greater interaction and more rapid 
development time are accomplished through use of unique techniques and capabilities 
provided within a VCE environment. 

• Effective information sharing and generation. The ability to easily share resources from 
remote sites is a critical component of a VCE environment. This allows all involved 
team members to access data, drawings, and documents to enhance design development 
and more quickly deal with specification changes. Such information sharing also 
provides an ability to evaluate the use of cutting edge technology early in the process and 
makes industry expert consultants more accessible. Sharing of information also enables 
team members to have a common understanding of all issues involved. 

• Improved communication. The application of VCE removes geographical constraints and 
reduces time lost in traveling. It facilitates an enriched communication between and 
among participants. Team members will interactively evaluate virtual prototypes of 
product designs and evaluate alternative scenarios. They will be able to make decisions 
quicker since all team members share the same information. 

• Improved development programs. VCEs will link physically dispersed teams for an 
integrated product and process development. This allows suppliers, users, and clients to 
provide feedback early in the engineering cycle, which enables team members to 
incorporate product lifecycle concerns. Such integration will also have an impact on the 
product quality. 
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In the following sections, we propose a system of systems or metasystems approach to e- 
engineering that can assist rapidly-organized project teams in meeting the challenges and needs 
of global engineering and manufacturing enterprises in virtual collaborative environments 
(VCEs). Applications of this approach to an ISAT case scenario are also described. Viewing 
enhanced distributed e-engineering environments as a metasystem provides several guiding 
principles from which to approach this problem. Systems of systems must be engineered in 
terms that provide effective design, deployment, operation, transformation, and evaluation. 
These new “higher order” systems must be focused on producing “systems of systems” 
performance as opposed to individual performance of subordinate systems. The design, 
deployment, operation, and transformation of higher level metasystems involve the integration 
multiple complex system processes to produce desirable results. These metasystems are 
themselves comprised of multiple autonomous embedded complex systems that can be diverse in 
technology, context, operation, geography, and conceptual frame. (Keating et. al., 2002). At the 
“metasystem” level, true optimization is a fallacy. Complex turbulent contexts and environments 
preclude optimization in the traditional systems engineering sense. Satisficing suggests that 
SoSE should focus on development of satisfactory solutions that are a continual refinement of 
the e-engineering environment. This perspective appreciates the continual evolution and 
tailoring of e-engineering system problem(s) requirements, boundaries, entities, and relationships 
throughout the a project’s e-engineering effort. 
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2. Concepts for the e-engineering project cycle 

In order to help address the above challenges for global engineering, an important research 
question is ‘what changes are required for rapidly-organized engineering teams to quickly 
assimilate and execute at high e-engineering performance levels and how can these changes be 
quickly implemented and sustained?’ In order to provide some foundation to address this 
question, it is useful to first discuss selected project characteristics and virtual team concepts. 
This discussion will also help answer the portion of the research question concerning what 
critical adaptation areas are useful for project teams to perform at high e-engineering levels. 

Project characteristics serve to partially define work conducted in virtual collaborative 
engineering environments. A project can be defined as ‘a temporary endeavor undertaken to 
create a (deliverable) unique product or service’ (PMI Standards Committee, 1996). Project 
phases are collectively known as the project life cycle, which defines a project’s beginning, 
phase sequencing, and end. 

Many models exist for project life cycles (Dorfman, 1977). One extensively used model is 
the waterfall model containing sequential phases of requirements, design, build, test, and 
integration. Each waterfall phase should be essentially complete before the next phase begins. 
This model encounters problems when project requirements do not remain stable following 
completion of the requirements phase. One approach to enhancing the waterfall model is the 
prototyping model, which makes use of system prototypes with selected functionality to help 
determine accurate requirements. These prototypes are developed using compressed waterfall 
sub-models early in the requirements phase of a traditional waterfall model. 

The spiral life cycle model (Boehm, 1988) is an innovation that permits combinations of 
conventional (e.g., waterfall) and enhanced (e.g., prototyping) to be used for various portions of 
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a project. Recently, the spiral model has been clarified by Boehm (2000) to capture its essence. 
Spiral development is a risk-driven process model generator with two main features. The first 
feature is a cyclic approach to incrementally grow a project’s degree of detinition while 
decreasing its degree of risk. The other main feature is the use of anchor point milestones to 
ensure stakeholder review and commitment during spiral cycles. Boehm has also clarified that 
the spiral model is not just a sequence of waterfall increments with activities following a single 
spiral sequence. On the contrary, the order of activities in the spiral is a guideline with the actual 
order of visiting or revisiting activities driven by ongoing project assessments. It is also 
important to emphasize that the spiral model is a process-oriented model where each cycle 
includes assessment and improvement of project processes as well as project deliverables. 

Spiral development has a focus concerning software projects (e.g., Muench, 1994), but can 
be applied more generally to project life cycles, including e-engineering projects. The spiral 
model’s emphasis on improving both project processes and deliverables fits well with the 
challenges of integrating e-engineering process improvements during the project life cycle. With 
the expected compression of project timelines and constrained budgets in advanced engineering 
environments, the spiral model’s cyclic development approach and embedded iterative risk 
assessments can accelerate e-engineering team development, accurate project requirements 
definition and decrease project uncertainty and risk. The anchor point milestones can serve to 
formalize progress concerning e-engineering performance levels, as well as the approval and 
hand-off of external deliverables. 


8 



6. E-engineering interactions, dynamics, and technology environment 

One critical aspect of e-engineering is for project teams to understand and apply the various 
types of distributed collaborative interactions. A general model of distributed collaboration 
dynamics is shown in Figure 2 (Dix et. al., 1998). A common environment is established and 
entities populate the environment, including project participants (e.g., team members and 
external stakeholders) and artifacts (e.g., documents and virtual or physical prototypes). 
Interactions can occur between participants and between participants and project artifacts. Direct 
communication interactions are conducted between participants using synchronous and 
asynchronous tools, including audio, video, and text messaging. Participants interact with 
project artifacts by controlling artifacts and receiving feedback using artifact tools. Participants 
also indirectly interact with each other through these artifact tools. Two forms of this indirect 
interaction are feedthrough and deixis. Feedthrough occurs when a participant’s manipulation of 
shared artifact objects is viewed by others (e.g., rotation of a 3D CAD design). Deixis occurs 
when referencing an artifact aspect to other project participants (e.g., pointing with a cursor to a 
feature of the CAD drawing). 



Figure 2. Distributed communication and work interactions. 

This distributed collaboration interaction model can be extended and viewed more 
analytically in terms of an object-oriented approach to project processes and interactions. 
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Object-oriented extensions could be especially useful for modeling distributed collaborative 
processes in specific project scenarios. One object-oriented approach to business modeling treats 
project processes as objects, as well as other entities of the application domain (Bider & 
Khomyakov, 1997). In terms of the above distributed project collaboration model, objects can 
include entities (such as virtual team members, data artifacts, and supporting collaboration tools), 
as well as the interactions (such as direct communication and data artifact interactions) between 
such entities. In this domain, relations between conceptual objects are as important as objects 
themselves and entity interactions are active, not passive. Such distributed collaboration objects 
are complex and dynamic and the properties of objects can be represented with the help of: 
history, events, and activities. History is the time-ordered sequence of all the previous states of 
objects. The time-ordered history is most important for objects that represent collaborative 
processes as it shows the evolution of the process in time. Events present additional information 
about transitions from one state of an object to another, including date, time, impacted objects, 
and event attributes. Activities represent distributed collaboration actions that take place in the 
project domain, like the various types of collaborative model interactions described above.Such a 
distributed communication and work interaction model is now enhanced to represent e- 
engineering interactions and associated technology areas to enable this interaction, as shown in 
Figure 3. The two main areas are user-centric tools, representing direct participant-to-participant 
communication and artifact-centric tools, representing participant-to-artifact interaction. User- 
centric tools can take both asynchronous (e.g., email) and synchronous (e.g. electronic 
conferences, video connections, audio, and text messaging) direct communication forms. 
Artifact-centric tools can include project management and scheduling applications, product and 
process simulation, and discipline-specific tools needed for various project deliverable scenarios. 
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The same categories of groupware interactions from Figure 2 also apply and are shown in Figure 
3. In order to conduct such interactions in a globally distributed environment, various 
technology layers are represented. A human-computer interaction (HCI) layer needs to exist 
between participants and the people or artifact-centric application tools. Such HCI technology 
includes computer input and display devices. Application interfaces are required to transfer 
communications and artifact data between various project applications. These interfaces involve 
translation protocols and data format standards. Network infrastructures need to exist to transmit 
interaction data to other distributed locations via local and wide area networks. A knowledge 
repository can also be incorporated to manage a complex project s artifacts, facilitating 
configuration management and quality control. 
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Figure 3. Virtual collaborative metasystem distributed interaction environment 


It is important for e-engineers to understand the above technology areas involved in 
distributed collaborative work environments. Critical issues in applying this technology include 


11 


defining user requirements, tool selection, network requirements, systems requirements, and 
emerging technology standards. Obtaining user requirements for collaborative tool use in 
projects can be difficult and requires personnel with a good technical understanding of 
collaborative tools as well as project tasks. The team leader and members must coordinate with 
information technology staff in developing these requirements. One option is to develop a 
“strawman” list and present this list to a group of team members for validation. With a 
completed list of requirements, tools are identified to meet the stated requirements from the 
existing collaborative toolset in the organization or adding additional tools. Network 
requirements also need to be taken into consideration, since deploying collaborative tools can 
have a severe impact on a network. In order to get the distributed collaboration infrastructure 
working (especially between organizations), firewall security issues need to be resolved as well 
as the environment’s impact on network bandwidth. System requirements for the e-engineering 
environment can include upgrade of hardware peripherals, including headsets and desktop 
cameras for videoconferencing. The existing technology infrastructure needs to be leveraged as 
much as possible, since a majority of required e-engineering capabilities can typically be met 
with existing tools. 

The issue of open standards is also important to understand when deploying and upgrading 
e-engineering infrastructures. Emerging standards for distributed collaboration include T. 120 
standards addressing real time data conferencing (audiographics), the H.323 standard addressing 
video (audiovisual) communication on local area networks, and the H.324 standard addressing 
video and audio communications over low bit rate connections such as modem connections. 
Even though these standards are being widely used, many tools still are using proprietary 
protocols and this can impact integration of these tools within a distributed project environment. 
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Virtual team and task dynamics 


Another aspect of e-engineering interactions deals with team and task dynamics during 
projects. The task performance of teams using e-engineering technology to communicate and 
collaborate can be viewed as a series of stages (McGrath, 1990). These task stages are 1) 
inception , 2) problem solving, 3) conflict resolution, and 4) execution. Inception involves 
defining project goals. The problem solving stage deals with development of solutions to project 
technical problems. Conflict resolution occurs when different points of view and approaches 
need to be reconciled. Also, different cultural and organizational perspectives could require 
resolution. Finally, execution involves performing project tasks and overcoming project and 
organizational barriers that inhibit performance. These task dynamic stages are not necessarily 
sequential and certain stages may not be required, depending on the project scenario and 
complexity. A team might go from inception directly to execution for more repeatable, 
prescriptive project scenarios or repeat iterations between problem solving and conflict 
resolution with difficult scenarios. Duarte & Snyder (1999) have identified four virtual team 
social dynamic stages, which parallel the above task dynamics. Social stage 1, interaction and 
inclusion , is where the team identifies and maps individual skills to project needs, establishes 
communication and work procedures, and develops initial plans. Social stage 2, position status 
and role definition, involves member role definition and status relationships. In social stage 3, 
allocation of resources and power, the team addresses allocation of resources and member power 
relationships. Social stage 4, interaction and participation, involves performance of 
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collaborative work including interaction, participation among the team members, and 
overcoming productivity barriers. 

Implementation issues 

Understanding and developing proficiency in the above aspects of virtual team interactions 
and dynamics are essential for attaining rapid, high performance levels in e-engineering 
environments. Collaborative interaction and technology areas, as well as task and social virtual 
team stages are critical e-engineering areas to understand, establish, and continually improve 
during the project life cycle. Initial assessments need to be made of team member global 
distribution, technology capability, and skill levels in distributed collaboration as well as relevant 
engineering-specific disciplines. Individuals and the entire team need to be trained in identified 
e-engineering skill and knowledge area deficiencies. Virtual collaborative functionality needs 
should be mapped to project activities and technology solutions identified to enable this 
capability. Collaboration technology, task, and social processes should be iteratively assessed 
and continually improved during the project life cycle. 


4. The model for e-engineering team adaptation (MeTA) 

Now that a foundation of literature has been discussed and e-engineering-related concepts 
identified, an initial framework is proposed, called the Model for e-engineering Team Adaptation 
(MeTA) to help improve the performance of such global engineering teams. As part of a word’s 
structure, meta can indicate change, (e.g., metachromatism — a change in an organism s color 
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caused by variation of physical conditions). Similarly, MeTA can be thought of as a framework 
of changes implemented to a project team’s dynamics, required by varying the team’s physical 
and information technology environment to virtual collaborative engineering. The model, which 
is pictured in Figure 4, uses an adaptation of the spiral software development approach (Boehm, 
1988), to integrate e-engineering process and product development activities. In order to quickly 
‘spin up’ to high project performance, the team conducts various e-engineering process and 
product-centric cycles. 

MeTA is a process-oriented model where each cycle includes assessment and improvement 
of project processes as well as project deliverables. Similar to other spiral models, each cycle of 
the model goes through actions portrayed as a quadrant. MeTA uses action categories of 
identify, plan, execute, and assess. The model has two main phases in the spiral itself: e- 
engineering basics and e- engineering application. As with the general spiral development 
model, MeTA is not just a sequence of waterfall increments with activities following a single 
spiral sequence. The order of activities in the spiral is a guideline with the actual order of 
visiting or revisiting activities driven by ongoing project assessments. In fact, project activities 
or sub-activities could be happening simultaneously in multiple MeTA cycles or phases. Anchor 
point milestones reviews are conducted in the assessment actions quadrant concerning e- 
engineering performance levels and external deliverable progress and hand-offs. 
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Figure 4. Model for e-engineering team adaptation (MeTA) phasing and sample ISAT Case 
activities. 


4 . 1. The e-engineering basics phase 

The focus of the MeTA basics phase deals with an engineering team quickly reaching 
proficiency in basic e- engineering process areas. In this area of the model, both individual and 
team e-engineering skill cycles are addressed. MeTA action quadrants of identify, plan, execute, 
and assess are used to bring the team to required skill proficiency. For both the individual and 
team skill cycles, proficiency assessments are initially conducted by the team leader or an 
external source. 
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E-engineering skill deficiency areas are identified and individual training is planned and 
executed to achieve proficiency in these areas. Individual training is then followed by individual 
qualification assessments to establish proficiency. Individual areas can include collaboration 
tool skills and virtual team process concepts, project management and scheduling, as well as 
engineering-discipline skills required for a specific project scenario. 

The team skill cycle also starts with initial proficiency assessments of the team’s e- 
engineering performance, by the team itself or by external evaluation. E-engineering team 
deficiency areas are identified and team training and exercises planned and executed to achieve 
proficiency in these areas. Team training is then followed by team qualification assessment to 
establish proficiency. Teaming skills include performing at proficient levels in virtual team task 
and social dynamics as well as working effectively using distributed synchronous and 
asynchronous collaboration tools. 

4.2. The e-engineering application phase 

In the second MeTA phase, application, the focus shifts to the team applying its e- 
engineering proficiency to system development or other deliverable goals. This does not mean 
that e-engineering process refinement activities are over, just that they are now focused on 
supporting project development goals. MeTA action quadrants of identify, plan, execute, and 
assess are now used to support iterative project deliverable development cycles. In Figure 4, the 
e-engineering application area is tailored to the ISAT case scenario, with vehicle closure, safety 
& reliability, operations, and cost & economics modeling and analysis cycles. 
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It should be stressed that MeTA cycles and activities should be tailored to specific e- 
engineering project scenarios, but there are certain MeTA ‘invariants that define the essence of 
this model. These invariants use Boehm s spiral software development model invariants 
(Boehm, 2000) as a start point. The first invariant is that MeTA is a process-driven model, 
concerned with a team’s e-engineering process improvement as well as deliverable task progress. 
As such, MeTA must contain phase areas directed at e-engineering process proficiency (e.g., 
basics phase) as well as deliverable progress (application phase). The second invariant is that 
MeTA is a risk-driven assessment model where iterative process and deliverable assessments 
determine the type and level of effort of upcoming activities. The sequence of spiral activities is 
just a guide. In reality, project activities or sub-activities in multiple cycles could happen 
simultaneously in multiple MeTA cycles or phases, depending on these project assessments. The 
third invariant is that MeTA contain anchor point milestones formalize progress concerning e- 
engineering performance levels, as well as the approval and hand-off of external deliverables 

6. e-Engineering applications to an ISAT case scenario 

NASA’s Inter-center Systems Analysis Team (ISAT) is conducting a pilot enhanced 
distributed engineering initiative. ISAT engineering analysis activities include individual 
assessment discipline teams conducting vehicle closure, safety and reliability, operations, and 
economic modeling, which are very sequentially interdependent. Following implementation of a 
semi-distributed environment and Product Data Management (PDM), several of the specific 
discipline activities became parallel in nature (Fletcher, 2001) with teams working independently 
before sharing model input and output parameters. Applications of the e-engineering approach 
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to the ISAT case scenario are now described. Figure 5 shows an e-Engineering Entity view of 
the ISAT case task environment, where entities include key participants (shown in yellow) and 
artifacts (shown in green) as described previously in Figure 3. Entities are organized in terms of 
analysis activities and general sequencing of these activities in the ISAT case scenario. 
Participants are also identified by team role and geographic NASA center location. 

This e-Engineering Entity view is then enhanced to an Entity/Interaction view shown in the 
diagonal and upper portions of Figure 6. This can be treated as a type of systems engineering 
functional or behavioral view of the distributed engineering environment. This Entity/Interaction 
view is necessary to capture before e-Engineering infrastructure and methodologies are tailored 
and implemented for a program or project scenario. This view drives the type of technology 
implementations that can meet team distributed functionality needs for projects and work 
packages. Both user and data-centric interactions shown in Figure 3, which are necessary for 
effective performance of distributed ISAT analysis tasks, are mapped to scenario user and 
artifact entities. User-centric communication interactions include audio, video, and messaging 
interaction channels between participant users and analysis sub-teams. Data-centric interactions 
include shared application control, viewing, referencing, storage of artifact files, and sharing of 
project files. In terms of the ISAT scenario, Figure 6 shows the need for user-centric interactions 
at the overall ISAT team level, but also at each analysis sub-team level. All participants should 
have the capability to conduct synchronous dialog with other sub-team member and dialog 
between sub-teams on an individual or group basis. Such dialog is more natural with 
synchronous video, audio, and instant messaging capabilities. Also shown is the need for data- 
centric interactions within and between sub-teams. Within sub-teams, users need to be able to 
control, store, reference, and view modeling and analysis applications. Between groups, for 


19 


viewing and referencing interaction channels as well as file sharing are needed for ISAT team 
activities, including model error checking, model parameter input and output between dependent 
model interfaces, and synthesis of analysis across models. 

Another logical view of e-Engineering entities and functional interactions is shown in Figure 
7, which has similar information in a matrix organization. Entities are organized along the 
diagonal, with e-Engineering interactions shown at matrix intersections for participant - 
participant and participant-artifact interactions. ISAT Team interaction requirements are shown 
along the top row. Clusters of sub-team requirements are also shown for between participant and 
model interactions. 

By comparing this functional view with current or proposed implementations, an e- 
Engineering impact analysis can be conducted to analyze the traceability of functional 
requirements to implementations. As an example, Figure 8 shows this integrated functional and 
implementation view using the observed implementation of the ISAT engineering environment. 
On the lower half of the view current e-Engineering technology and processes can be identified 
for team, participant, and data model interactions. User-centric communication was dominated 
by an overall ISAT team room videoconferencing between centers. Data-centric communication 
employed an enterprise product data model (PDM) solution, which allowed integration between 
standalone analysis models, data storage, and file sharing. An e-Engineering impact analysis of 
the ISAT case highlights priorities for improvement for future distributed environments. 

ISAT e-Engineering traceability issue #1 deals with the inadequacy of room 
videoconferencing to meet user-centric communication needs. ISAT consists of multiple teams 
and a group audio and video channel is inadequate for user communications by individuals 
within and between teams. Possible technology solutions include multi-cast desktop 
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videoconferencing, similar to the e-Engineering classroom at Old Dominion University, which 
allows individual and group audio, video, and application sharing. Such tools should include the 
capability to interactively view any conference participant (whether speaking or listening), have 
a list of current participant names, allow file transfer, and whiteboarding. 

ISAT e-Engineering traceability issue #2 also deals with the inadequacy of room 
videoconferencing to meet user-centric communication needs. As shown in the functional view, 
multiple conferences between participants and sub-teams could be required at the same time. 
With a one channel room videoconferencing solution, only one session can occur at a time within 
a collaborative area. Desktop videoconferencing or other multichannel solutions can allow tor 
parallel distributed dialogs to occur, which maps better into the parallel nature of the ISAT 
workflow. Such simultaneous multiple conferences are possible within a single center s 
collaborative area, or from individual participant desktops. 

ISAT e-Engineering traceability issue #3 deals with the ability remotely view, reference, and 
control data model applications. If analysis tasks are conducted on standalone computers, 
without the ability for application sharing, the collaboration workflow becomes more like 
information sharing, where model input parameter values and results are “throw over the wall” 
and transferred to the next analyst or sub-team. Application sharing is essential to remotely 
view, reference, and control data model applications between multiple team members, which can 
help in process and error checking, as well as compress analysis times and resources 
requirements. 


6. e-Engineering ISAT case analysis conclusion 
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This section of the ISAT case research was conducted by Dr. David Dryer at Old Dominion 
University and contains a preliminary system of systems engineering approach for the iterative 
design, implementation, and improvement of e-Engineering project teams. This approach 
includes use of the MeTA model for quickly increasing and maintaining basic and applied e- 
Engineering proficiency. The approach also outlines a systems engineering process to assist in 
identifying distributed collaboration interaction requirements for a particular task environment, 
designing infrastructure solutions, and graphically assessing the traceability impact of current 
and proposed environments. 
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Figure 6. Enhanced Entity and e-Engineering Interactions (User and Data Centric) View of ISAT Case activities. 




















Figure 7. ISAT Case e-Engineering Entity/Interaction Functional View. 
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Figure 8. ISAT Case e-Engineering Functional and Current Implementation Views. 
































References 


Bider, I. and Khomyakov, M., 1997, One practical object-oriented model of business process. 

In Klimov H., Rumpe B., Simmonds I., (Eds.), OOPSLA’97 Workshop on Object-oriented 
Behavioral Semantics. Institute Fur Informatic. Technische Universitat Munchen, TUM- 
19737, 25- 31. 

Boehm, B., 1988, A spiral model of software development and enhancement. Computer, 21(5), 
61-72. 

Boehm, B., 2000, Spiral development: experience, principles, and refinements. Special Report 
CMU/SEI-00-SR-08, ESC-SR-00-08, W. J. Hansen (Ed.), Pittsburgh, PA: Carnegie 
Mellon University. 

Dix, A., Finaly, J., Abowd, G„ & Beale, R., 1998, Human-Computer Interaction (2d edition), 
London: Prentice Hall Europe. 

Dorfman, M., 1977, Requirements engineering, In Software Requirements Engineering. Second 
Edition, R. H. Thayer and M. Dorfman, (Eds.), Los Alamitos, CA: IEEE Computer 
Society Press, 7-22. 

Duarte, D. L. and Snyder, N. T., 1999, Mastering virtual teams. San Francisco: Jossey-Bass 
Publishers. 

Fletcher, T. D. (2001). Team task analysis of ISAT - Inter-center Systems Analysis Team during 
the August 2001 workshop as viewed from LaRC. Prepared for National Aeronautics and 
Space Administration. 


26 


Keating, C„ Rogers, R„ Unal, R., Dryer, D, Sousa-Poza, A, Safford, R., Peterson, W. (2002) 
Systems Of Systems Engineering. Proceedings of 23d Annual American Society for 
Engineering Management (ASEM) Conference (October 4-7), Tampa, FL. 

McGrath, J., 1990, Time matters in teams, In Intellectual Teamwork: Social and Technical 
Foundations of Cooperative Work. J. Galegher, R.E. Kraut, and C. Egido (Eds.), 

Hillsdale, N.J.: Erlbaum. 

Mills, A., 1998, Collaborative Engineering and the Internet. Society of Manufacturing 
Engineers, Dearborn, MI. 

Muench, D., 1994, The Sybase development framework. Oakland, CA: Sybase Inc. 

National Research Council, 1999, Advanced Engineering Environments: Achieving the Vision: 
Phase 1, Washington, D.C.: National Academy Press. 

National Research Council, 2000, Advanced Engineering Environments: Design in the new 
Millennium: Phase 2, Washington, D.C.; National Academy Press. 

PMI Standards Committee, 1996, A Guide to the Project Management Body of Knowledge. 
Project Management Institute, William R. Duncan, Director of Standards, Newtown 

Square, PA. 


27 



